An Overview of Key Ideas

讲座摘要（lecture summary）

An overview of key ideas

This is an overview of linear algebra given at the start of a course on the mathematics of engineering.

$$\text{Linear algebra progresses from vectors to matrices to subspaces.}$$

Vectors

What do you do with vectors? Take combinations.

We can multiply vectors by scalars, add, and subtract. Given vectors $\bm{u}$, $\bm{v}$ and $\bm{w}$ we can form the linear combination $x_1\bm{u} + x_2\bm{v} + x_3\bm{w} = b$.

An example in $\mathbb{R}^3$ would be:

$$\bm{u} = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \bm{v} = \begin{bmatrix} 0 \\ 1 \\ -1 \end{bmatrix}, \bm{u} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$$

The collection of all multiples of $\bm{u}$ forms a line through the origin. The collection of all multiples of $\bm{v}$ forms another line. The collection of all combinations of $\bm{u}$ and $\bm{v}$ forms a plane. Taking all combinations of some vectors creates a subspace.

We could continue like this, or we can use a matrix to add in all multiples of $\bm{w}$.

Matrices

Create a matrix $A$ with vectors $\bm{u}$, $\bm{v}$ and w in its columns:

$$A = \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & -1 & 1 \end{bmatrix}$$

The product:

$$A\bm{x} = \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & -1 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2\\ x_3 \end{bmatrix} = \begin{bmatrix} x_1 \\ -x_1 + x_2 \\ -x_2 + x_3 \end{bmatrix}$$

equals the sum $x_1\bm{u} + x_2\bm{v} + x_3\bm{w} = b$. The product of a matrix and a vector is a combination of the columns of the matrix. (This particular matrix $A$ is a difference matrix because the components of $A\bm{x}$ are differences of the components of that vector.)

When we say $x_1\bm{u} + x_2\bm{v} + x_3\bm{w} = b$ we’re thinking about multiplying numbers by vectors; when we say $A\bm{x} = \bm{b}$ we’re thinking about multiplying a matrix (whose columns are $\bm{u}$, $\bm{v}$ and $\bm{w}$) by the numbers. The calculations are the same, but our perspective has changed.

For any input vector $\bm{x}$, the output of the operation “multiplication by $A$” is some vector $\bm{b}$:

$$A\begin{bmatrix} 1 \\ 4\\ 9 \end{bmatrix} = \begin{bmatrix} 1 \\ 3\\ 5 \end{bmatrix}$$

A deeper question is to start with a vector b and ask “for what vectors x does $A\bm{x} = \bm{b}$?” In our example, this means solving three equations in three unknowns. Solving:

$$A\bm{x} = \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & -1 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2\\ x_3 \end{bmatrix} = \begin{bmatrix} x_1 \\ -x_1 + x_2 \\ -x_2 + x_3 \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2\\ b_3 \end{bmatrix}$$

is equivalent to solving:

$$\begin{array}{rcl} x_1 & = & b_1 \\ x_2 - x_1 & = & b_2 \\ x_3 - x_2& = & b_3 \end{array}$$

We see that $x_1 = b_1$ and so $x_2$ must equal $b_1 + b_2$. In vector form, the solution is:

$$\begin{bmatrix} x_1 \\ x_2\\ x_3 \end{bmatrix} = \begin{bmatrix*}[r] b_1 \\ b_1 + b_2\\ b_1 + b_2 + b_3 \end{bmatrix*}$$

But this just says:

$$\bm{x} = \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0\\ 1 & 1 & 1 \end{bmatrix} \begin{bmatrix} b_1 \\ b_2\\ b_3 \end{bmatrix}$$

or $\bm{x} = A^{−1}\bm{b}$. If the matrix $A$ is invertible, we can multiply on both sides by $A^{−1}$ to find the unique solution $\bm{x}$ to $A\bm{x} = \bm{b}$. We might say that A represents a transform $\bm{x} \to \bm{b}$ that has an inverse transform $\bm{b} \to \bm{x}$.

In particular, if $\bm{b}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$ then $\bm{x}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$

The second example has the same columns $\bm{u}$ and $\bm{v}$ and replaces column vector $\bm{w}$:

$$C = \begin{bmatrix} 1 & 0 & -1 \\ -1 & 1 & 0\\ 0 & -1 & 1 \end{bmatrix}$$

Then:

$$C\bm{x} = \begin{bmatrix} 1 & 0 & -1 \\ -1 & 1 & 0\\ 0 & -1 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2\\ x_3 \end{bmatrix} = \begin{bmatrix} x_1 - x_3 \\ x_2 - x_1 \\ x_3 - x_2 \end{bmatrix}$$

and our system of three equations in three unknowns becomes circular.

Where before $A\bm{x} = 0$ implied $\bm{x} = 0$, there are non-zero vectors $\bm{x}$ for which $C\bm{x} = 0$. For any vector $\bm{x}$ with $x_1 = x_2 = x_3, C\bm{x} = 0$. This is a significant difference; we can’t multiply both sides of $C\bm{x} = 0$ by an inverse to find a nonzero solution $\bm{x}$.

The system of equations encoded in $C\bm{x} = \bm{b}$ is:

$$\begin{array}{rcl} x_1 - x_3 & = & b_1 \\ x_2 - x_1 & = & b_2 \\ x_3 - x_2& = & b_3 \end{array}$$

If we add these three equations together, we get:

$$0 = b_1 + b_2 + b_3$$

This tells us that $C\bm{x} = \bm{b}$ has a solution $\bm{x}$ only when the components of b sum to 0. In a physical system, this might tell us that the system is stable as long as the forces on it are balanced.

Subspaces

Geometrically, the columns of $C$ lie in the same plane (they are dependent; the columns of $A$ are independent). There are many vectors in $\mathbb{R}^3$ which do not lie in that plane. Those vectors cannot be written as a linear combination of the columns of $C$ and so correspond to values of $\bm{b}$ for which $C\bm{x} = \bm{b}$ has no solution $\bm{x}$. The linear combinations of the columns of $C$ form a two dimensional subspace of $\mathbb{R}^3$.

This plane of combinations of $\bm{u}$, $\bm{v}$ and $\bm{w}$ can be described as “all vectors $C\bm{x}$”. But we know that the vectors $\bm{b}$ for which $C\bm{x} = \bm{b}$ satisfy the condition $b_1 + b_2 + b_3 = 0$. So the plane of all combinations of $\bm{u}$ and $\bm{v}$ consists of all vectors whose components sum to $0$.

If we take all combinations of:

$$\bm{u} = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \bm{v} = \begin{bmatrix} 0 \\ 1 \\ -1 \end{bmatrix}, \text{and}\> \bm{w} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$$

we get the entire space $\mathbb{R}^3$; the equation $A\bm{x} = \bm{b}$ has a solution for every $\bm{b}$ in $\mathbb{R}^3$. We say that $\bm{u}$, $\bm{v}$ and $\bm{w}$ form a basis for $\mathbb{R}^3$.

A basis for $\mathbb{R}^n$ is a collection of $n$ independent vectors in $\mathbb{R}^n$. Equivalently, a basis is a collection of n vectors whose combinations cover the whole space. Or, a collection of vectors forms a basis whenever a matrix which has those vectors as its columns is invertible.

A vector space is a collection of vectors that is closed under linear combinations. A subspace is a vector space inside another vector space; a plane through the origin in $\mathbb{R}^3$ is an example of a subspace. A subspace could be equal to the space it’s contained in; the smallest subspace contains only the zero vector.

The subspaces of $\mathbb{R}^3$ are:

• the origin,
• a line through the origin,
• a plane through the origin,
• all of $\mathbb{R}^3$.

Conclusion

When you look at a matrix, try to see “what is it doing?”

Matrices can be rectangular; we can have seven equations in three unknowns. Rectangular matrices are not invertible, but the symmetric, square
matrix $A^TA$ that often appears when studying rectangular matrices may be invertible.