Vector Spaces, Subspaces, Independence, Basis and Dimension

Vector spaces

A vector space over $\mathbb R$ is a set $V$ with addition and scalar multiplication satisfying the familiar rules: closure under both operations, commutativity and associativity of addition, a zero vector, additive inverses, distributivity ($a(\mathbf u + \mathbf v) = a\mathbf u + a\mathbf v$, $(a + b)\mathbf v = a\mathbf v + b\mathbf v$), $a(b\mathbf v) = (ab)\mathbf v$, and $1\mathbf v = \mathbf v$.

Subspaces

A non-empty subset $W \subseteq V$ is a subspace iff it is closed under linear combinations:

$$\mathbf u, \mathbf v \in W,\ a, b \in \mathbb R \implies a\mathbf u + b\mathbf v \in W.$$

Every subspace contains $\mathbf 0$ — the fastest way to rule a set out. Subspaces of $\mathbb R^3$ are: $\{\mathbf 0\}$, lines through the origin, planes through the origin, and $\mathbb R^3$ itself.

Span and linear combinations

$\operatorname{span}\{\mathbf v_1, \dots, \mathbf v_k\} = \{c_1\mathbf v_1 + \dots + c_k \mathbf v_k\}$ is the smallest subspace containing the vectors. $\mathbf b \in \operatorname{span}\{\dots\}$ iff the system $[\mathbf v_1 \cdots \mathbf v_k]\,\mathbf c = \mathbf b$ is consistent.

Linear dependence and independence

Vectors are linearly independent if

$$c_1\mathbf v_1 + c_2 \mathbf v_2 + \dots + c_k \mathbf v_k = \mathbf 0 \implies c_1 = c_2 = \dots = c_k = 0.$$

Otherwise some vector is a combination of the others (dependent).

How to test:

1. Put the vectors as columns of $A$; they are independent iff $\operatorname{rank}(A) = k$ iff $A\mathbf c = \mathbf 0$ has only the trivial solution.
2. For $n$ vectors in $\mathbb R^n$: independent iff $\det A \ne 0$.
3. Shortcuts: any set containing $\mathbf 0$ is dependent; more than $n$ vectors in $\mathbb R^n$ are dependent; two vectors are dependent iff one is a multiple of the other; non-zero mutually orthogonal vectors are independent.

Basis and dimension

A basis is a linearly independent spanning set. Every basis of a space has the same number of vectors, the dimension.

• In an $n$-dimensional space, any $n$ independent vectors form a basis, and any $n$ spanning vectors form a basis.
• Any independent set can be extended to a basis; any spanning set can be reduced to one.
• Coordinates: each $\mathbf v$ is a unique combination of basis vectors; the coefficients are its coordinates in that basis. If $B$ has the basis vectors as columns, $[\mathbf v]_B = B^{-1}\mathbf v$.

Sums and intersections

For subspaces $U, W$:

• $U \cap W$ and $U + W$ are subspaces; $U \cup W$ generally is not.
• $\dim(U + W) = \dim U + \dim W - \dim(U\cap W)$.
• If $U \cap W = \{\mathbf 0\}$, the sum is direct, $U \oplus W$, and every vector splits uniquely as $\mathbf u + \mathbf w$.

Inner products, norms and orthogonality

The standard inner product $\langle \mathbf u, \mathbf v\rangle = \mathbf u^\top \mathbf v = \sum u_i v_i$ gives:

• Norm: $\lVert\mathbf v\rVert = \sqrt{\mathbf v^\top \mathbf v}$; other norms: $\lVert\mathbf v\rVert_1 = \sum\lvert v_i\rvert$, $\lVert\mathbf v\rVert_\infty = \max \lvert v_i\rvert$.
• Angle: $\cos\theta = \dfrac{\mathbf u^\top\mathbf v}{\lVert\mathbf u\rVert\,\lVert\mathbf v\rVert}$ — the cosine similarity used in ML.
• Cauchy–Schwarz: $\lvert\mathbf u^\top\mathbf v\rvert \le \lVert\mathbf u\rVert\,\lVert\mathbf v\rVert$; triangle inequality $\lVert\mathbf u + \mathbf v\rVert \le \lVert\mathbf u\rVert + \lVert\mathbf v\rVert$.
• Orthogonal vectors: $\mathbf u^\top\mathbf v = 0$. An orthonormal set is orthogonal with unit vectors. In an orthonormal basis, coordinates are just dot products: $c_i = \mathbf q_i^\top \mathbf v$.
• Orthogonal complement: $W^\perp = \{\mathbf v : \mathbf v^\top \mathbf w = 0\ \forall \mathbf w \in W\}$, with $\dim W + \dim W^\perp = n$.

Gram–Schmidt

Turns an independent set $\mathbf a_1, \dots, \mathbf a_k$ into an orthonormal set spanning the same space:

$$\mathbf u_j = \mathbf a_j - \sum_{i<j}(\mathbf q_i^\top \mathbf a_j)\,\mathbf q_i, \qquad \mathbf q_j = \frac{\mathbf u_j}{\lVert\mathbf u_j\rVert}.$$

Each step subtracts the projections onto the earlier directions (see projections). In matrix form this is the QR decomposition $A = QR$.

Quick revision