Matrix algebra and its rules (transpose, inverse, trace), symmetric and skew-symmetric matrices, orthogonal matrices, idempotent and projection matrices, nilpotent and involutory matrices, permutation matrices, and partitioned (block) matrices with block multiplication, block determinants and the Schur complement.
Matrix algebra essentials
For an m × n m\times n m × n A A A n × p n \times p n × p B B B ( A B ) i j = ∑ k a i k b k j (AB)_{ij} = \sum_k a_{ik}b_{kj} ( A B ) ij = ∑ k a ik b k j not commutative .
Useful views of a product:
Column view : A x = x 1 a 1 + ⋯ + x n a n A\mathbf x = x_1\mathbf a_1 + \dots + x_n\mathbf a_n A x = x 1 a 1 + ⋯ + x n a n A A A Outer-product view : A B = ∑ k ( column k of A ) ( row k of B ) AB = \sum_k (\text{column } k \text{ of } A)(\text{row } k \text{ of } B) A B = ∑ k ( column k of A ) ( row k of B )
Rule Transpose ( A ⊤ ) ⊤ = A (A^\top)^\top = A ( A ⊤ ) ⊤ = A ( A B ) ⊤ = B ⊤ A ⊤ (AB)^\top = B^\top A^\top ( A B ) ⊤ = B ⊤ A ⊤ ( A + B ) ⊤ = A ⊤ + B ⊤ (A + B)^\top = A^\top + B^\top ( A + B ) ⊤ = A ⊤ + B ⊤ Inverse ( A B ) − 1 = B − 1 A − 1 (AB)^{-1} = B^{-1}A^{-1} ( A B ) − 1 = B − 1 A − 1 ( A ⊤ ) − 1 = ( A − 1 ) ⊤ (A^\top)^{-1} = (A^{-1})^\top ( A ⊤ ) − 1 = ( A − 1 ) ⊤ ( c A ) − 1 = c − 1 A − 1 (cA)^{-1} = c^{-1}A^{-1} ( c A ) − 1 = c − 1 A − 1 2 × 2 2\times 2 2 × 2 ( a b c d ) − 1 = 1 a d − b c ( d − b − c a ) \begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1} = \dfrac{1}{ad - bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix} ( a c b d ) − 1 = a d − b c 1 ( d − c − b a ) Trace tr ( A + B ) = tr A + tr B \operatorname{tr}(A + B) = \operatorname{tr}A + \operatorname{tr}B tr ( A + B ) = tr A + tr B tr ( A B ) = tr ( B A ) \operatorname{tr}(AB) = \operatorname{tr}(BA) tr ( A B ) = tr ( B A ) tr ( A B C ) = tr ( C A B ) \operatorname{tr}(ABC) = \operatorname{tr}(CAB) tr ( A B C ) = tr ( C A B ) Trace & eigenvalues tr A = ∑ λ i \operatorname{tr}A = \sum \lambda_i tr A = ∑ λ i
Common trap
tr ( A B ) = tr ( B A ) \operatorname{tr}(AB) = \operatorname{tr}(BA) tr ( A B ) = tr ( B A ) tr ( A B C ) ≠ tr ( B A C ) \operatorname{tr}(ABC) \ne \operatorname{tr}(BAC) tr ( A B C ) = tr ( B A C ) cyclic shifts are allowed. Similarly ( A + B ) 2 = A 2 + A B + B A + B 2 (A + B)^2 = A^2 + AB + BA + B^2 ( A + B ) 2 = A 2 + A B + B A + B 2 A 2 + 2 A B + B 2 A^2 + 2AB + B^2 A 2 + 2 A B + B 2 A B = B A AB = BA A B = B A
Catalogue of special matrices
Type Definition Key properties Diagonal a i j = 0 a_{ij} = 0 a ij = 0 i ≠ j i \ne j i = j eigenvalues = diagonal entries; products commute Triangular zeros below (upper) or above (lower) the diagonal det \det det Symmetric A ⊤ = A A^\top = A A ⊤ = A real eigenvalues, orthogonal eigenvectors (spectral theorem) Skew-symmetric A ⊤ = − A A^\top = -A A ⊤ = − A zero diagonal; eigenvalues purely imaginary or 0; det = 0 \det = 0 det = 0 n n n Orthogonal Q ⊤ Q = Q Q ⊤ = I Q^\top Q = QQ^\top = I Q ⊤ Q = Q Q ⊤ = I Q − 1 = Q ⊤ Q^{-1} = Q^\top Q − 1 = Q ⊤ det = ± 1 \det = \pm1 det = ± 1 ∣ λ ∣ = 1 \lvert\lambda\rvert = 1 ∣ λ ∣ = 1 Idempotent A 2 = A A^2 = A A 2 = A λ ∈ { 0 , 1 } \lambda \in \{0,1\} λ ∈ { 0 , 1 } I − A I - A I − A Projection (orthogonal) P 2 = P P^2 = P P 2 = P P ⊤ = P P^\top = P P ⊤ = P projects onto col ( P ) \operatorname{col}(P) col ( P ) I − P I - P I − P Nilpotent A k = 0 A^k = 0 A k = 0 k k k all eigenvalues 0; det = tr = 0 \det = \operatorname{tr} = 0 det = tr = 0 Involutory A 2 = I A^2 = I A 2 = I A − 1 = A A^{-1} = A A − 1 = A λ = ± 1 \lambda = \pm 1 λ = ± 1 Permutation rows of I I I orthogonal; det = ± 1 \det = \pm 1 det = ± 1 Positive definite symmetric, x ⊤ A x > 0 \mathbf x^\top A\mathbf x > 0 x ⊤ A x > 0 x ≠ 0 \mathbf x \ne 0 x = 0 all λ > 0 \lambda > 0 λ > 0 quadratic forms
Symmetric and skew-symmetric parts
Every square matrix splits uniquely as A = 1 2 ( A + A ⊤ ) ⏟ symmetric + 1 2 ( A − A ⊤ ) ⏟ skew A = \underbrace{\tfrac12(A + A^\top)}_{\text{symmetric}} + \underbrace{\tfrac12(A - A^\top)}_{\text{skew}} A = symmetric 2 1 ( A + A ⊤ ) + skew 2 1 ( A − A ⊤ ) x ⊤ A x = x ⊤ ( A + A ⊤ 2 ) x \mathbf x^\top A\mathbf x = \mathbf x^\top\left(\tfrac{A + A^\top}{2}\right)\mathbf x x ⊤ A x = x ⊤ ( 2 A + A ⊤ ) x
Orthogonal matrices
The columns (and rows) form an orthonormal basis. Orthogonal matrices represent rotations (det = 1 \det = 1 det = 1 det = − 1 \det = -1 det = − 1
R θ = ( cos θ − sin θ sin θ cos θ ) , H = I − 2 u u ⊤ ( ∥ u ∥ = 1 ) (Householder reflection) . R_\theta = \begin{pmatrix}\cos\theta & -\sin\theta\\ \sin\theta & \cos\theta\end{pmatrix}, \qquad H = I - 2\mathbf u\mathbf u^\top\ (\lVert \mathbf u\rVert = 1) \text{ (Householder reflection)}. R θ = ( cos θ sin θ − sin θ cos θ ) , H = I − 2 u u ⊤ (∥ u ∥ = 1 ) (Householder reflection) . Because ∥ Q x ∥ = ∥ x ∥ \lVert Q\mathbf x\rVert = \lVert\mathbf x\rVert ∥ Q x ∥ = ∥ x ∥
Idempotent and projection matrices
Every idempotent matrix is a projection (onto its column space, along its null space). It is an orthogonal projection iff it is also symmetric. In statistics:
the hat matrix H = X ( X ⊤ X ) − 1 X ⊤ H = X(X^\top X)^{-1}X^\top H = X ( X ⊤ X ) − 1 X ⊤ y \mathbf y y y ^ = H y \hat{\mathbf y} = H\mathbf y y ^ = H y tr ( H ) \operatorname{tr}(H) tr ( H )
the residual maker I − H I - H I − H tr ( I − H ) = n − p \operatorname{tr}(I - H) = n - p tr ( I − H ) = n − p
the centering matrix C = I − 1 n 11 ⊤ C = I - \frac1n\mathbf 1\mathbf 1^\top C = I − n 1 1 1 ⊤ rank ( C ) = n − 1 \operatorname{rank}(C) = n - 1 rank ( C ) = n − 1
Worked example
Projection onto the line spanned by a = ( 1 , 2 ) \mathbf a = (1, 2) a = ( 1 , 2 ) P = a a ⊤ a ⊤ a = 1 5 ( 1 2 2 4 ) P = \dfrac{\mathbf a\mathbf a^\top}{\mathbf a^\top\mathbf a} = \dfrac15\begin{pmatrix}1&2\\2&4\end{pmatrix} P = a ⊤ a a a ⊤ = 5 1 ( 1 2 2 4 ) P 2 = P P^2 = P P 2 = P tr P = 1 = rank P \operatorname{tr}P = 1 = \operatorname{rank}P tr P = 1 = rank P a \mathbf a a ( 2 , − 1 ) (2,-1) ( 2 , − 1 )
Partitioned (block) matrices
Split a matrix into sub-matrices; operations work blockwise when the shapes are compatible:
( A B C D ) ( E F G H ) = ( A E + B G A F + B H C E + D G C F + D H ) . \begin{pmatrix}A & B\\ C & D\end{pmatrix}\begin{pmatrix}E & F\\ G & H\end{pmatrix} = \begin{pmatrix}AE + BG & AF + BH\\ CE + DG & CF + DH\end{pmatrix}. ( A C B D ) ( E G F H ) = ( A E + B G C E + D G A F + B H C F + D H ) . Property Formula Transpose ( A B C D ) ⊤ = ( A ⊤ C ⊤ B ⊤ D ⊤ ) \begin{pmatrix}A&B\\C&D\end{pmatrix}^\top = \begin{pmatrix}A^\top & C^\top\\ B^\top & D^\top\end{pmatrix} ( A C B D ) ⊤ = ( A ⊤ B ⊤ C ⊤ D ⊤ ) Block diagonal det = det A det D \det = \det A\det D det = det A det D = diag ( A − 1 , D − 1 ) = \operatorname{diag}(A^{-1}, D^{-1}) = diag ( A − 1 , D − 1 ) Block triangular det ( A B 0 D ) = det A ⋅ det D \det\begin{pmatrix}A & B\\ 0 & D\end{pmatrix} = \det A \cdot \det D det ( A 0 B D ) = det A ⋅ det D Schur complement (A A A det ( A B C D ) = det A ⋅ det ( D − C A − 1 B ) \det\begin{pmatrix}A&B\\C&D\end{pmatrix} = \det A \cdot \det(D - CA^{-1}B) det ( A C B D ) = det A ⋅ det ( D − C A − 1 B )
The block inverse (with Schur complement S = D − C A − 1 B S = D - CA^{-1}B S = D − C A − 1 B
( A B C D ) − 1 = ( A − 1 + A − 1 B S − 1 C A − 1 − A − 1 B S − 1 − S − 1 C A − 1 S − 1 ) . \begin{pmatrix}A&B\\C&D\end{pmatrix}^{-1} = \begin{pmatrix}A^{-1} + A^{-1}BS^{-1}CA^{-1} & -A^{-1}BS^{-1}\\ -S^{-1}CA^{-1} & S^{-1}\end{pmatrix}. ( A C B D ) − 1 = ( A − 1 + A − 1 B S − 1 C A − 1 − S − 1 C A − 1 − A − 1 B S − 1 S − 1 ) . Note
Schur complements appear in conditioning multivariate normals: if ( X Y ) \begin{pmatrix}X\\Y\end{pmatrix} ( X Y ) ( Σ X X Σ X Y Σ Y X Σ Y Y ) \begin{pmatrix}\Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY}\end{pmatrix} ( Σ X X Σ Y X Σ X Y Σ Y Y ) Cov ( Y ∣ X ) = Σ Y Y − Σ Y X Σ X X − 1 Σ X Y \operatorname{Cov}(Y\mid X) = \Sigma_{YY} - \Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} Cov ( Y ∣ X ) = Σ Y Y − Σ Y X Σ X X − 1 Σ X Y
Quick revision
Practice questions GATE-style questions: 4 one-mark and 6 two-mark. Wrong MCQs cost ⅓ of their marks in the real exam; MSQ and NAT have no negative marking. Attempt each before opening the solution.
Q1 MSQ 2 marks
A A A n × n n \times n n × n idempotent matrix (A 2 = A A^2 = A A 2 = A A ≠ I A \ne I A = I
One or more options are correct.
A Every eigenvalue of A A A B I − A I - A I − A C rank ( A ) = trace ( A ) \operatorname{rank}(A) = \operatorname{trace}(A) rank ( A ) = trace ( A ) D A A A Check answer Show solution
Q2 NAT 1 mark
X X X 5 × 2 5 \times 2 5 × 2 P = X ( X ⊤ X ) − 1 X ⊤ P = X(X^\top X)^{-1}X^\top P = X ( X ⊤ X ) − 1 X ⊤ P P P
Check answer Show solution
Q3 MCQ 1 mark
If Q Q Q det Q \det Q det Q
A always 1 B either 1 or −1 C any non-zero real number D 0
Check answer Show solution
Q4 NAT 2 marks
M = ( A B 0 D ) M = \begin{pmatrix} A & B \\ 0 & D \end{pmatrix} M = ( A 0 B D ) 5 × 5 5 \times 5 5 × 5 A A A 2 × 2 2\times 2 2 × 2 det A = 3 \det A = 3 det A = 3 D D D 3 × 3 3\times3 3 × 3 det D = 4 \det D = 4 det D = 4 B B B 2 × 3 2\times 3 2 × 3 det M \det M det M
Check answer Show solution
Q5 MCQ 1 mark
For invertible n × n n\times n n × n A A A B B B ( A B ) − 1 (AB)^{-1} ( A B ) − 1
A A − 1 B − 1 A^{-1}B^{-1} A − 1 B − 1 B B − 1 A − 1 B^{-1}A^{-1} B − 1 A − 1 C ( B A ) − 1 (BA)^{-1} ( B A ) − 1 D A − 1 + B − 1 A^{-1} + B^{-1} A − 1 + B − 1 Check answer Show solution
Q6 NAT 2 marks
The matrix Q = ( 0.6 − 0.8 a 0.6 ) Q = \begin{pmatrix} 0.6 & -0.8 \\ a & 0.6 \end{pmatrix} Q = ( 0.6 a − 0.8 0.6 ) a a a
Check answer Show solution
Q7 MSQ 2 marks
For an arbitrary real n × n n \times n n × n A A A
One or more options are correct.
A A + A ⊤ A + A^\top A + A ⊤ B A − A ⊤ A - A^\top A − A ⊤ C A ⊤ A A^\top A A ⊤ A D A A ⊤ = A ⊤ A AA^\top = A^\top A A A ⊤ = A ⊤ A Check answer Show solution
Q8 NAT 2 marks
The centering matrix C = I 4 − 1 4 11 ⊤ C = I_4 - \frac14 \mathbf 1\mathbf 1^\top C = I 4 − 4 1 1 1 ⊤ 1 \mathbf 1 1 R 4 \mathbb R^4 R 4
Check answer Show solution
Q9 MCQ 2 marks
For any p × q p \times q p × q B B B ( I p B 0 I q ) \begin{pmatrix} I_p & B \\ 0 & I_q \end{pmatrix} ( I p 0 B I q )
A ( I p B 0 I q ) \begin{pmatrix} I_p & B \\ 0 & I_q \end{pmatrix} ( I p 0 B I q ) B ( I p − B 0 I q ) \begin{pmatrix} I_p & -B \\ 0 & I_q \end{pmatrix} ( I p 0 − B I q ) C ( I p 0 − B ⊤ I q ) \begin{pmatrix} I_p & 0 \\ -B^\top & I_q \end{pmatrix} ( I p − B ⊤ 0 I q ) D it is not invertible in general Check answer Show solution
Q10 NAT 1 mark
For A = ( 1 2 3 4 ) A = \begin{pmatrix}1 & 2\\3&4\end{pmatrix} A = ( 1 3 2 4 ) B = ( 0 1 1 0 ) B = \begin{pmatrix}0&1\\1&0\end{pmatrix} B = ( 0 1 1 0 ) trace ( A B ) \operatorname{trace}(AB) trace ( A B )
Check answer Show solution