Mathematics 3
Advanced Linear Algebra — Complete Course Summary
Orthogonality & Least Squares, Diagonalization, Complex Eigenvalues,
the Spectral Theorem, Quadratic Forms, and Singular Value Decomposition
Lecture Notes, Step-by-Step Solution Guides &
Worked Exam Exercises
This document is an independent study summary and is not affiliated with, endorsed by, or a substitute for
any university’s official course materials.
,Mathematics 3 — Advanced Linear Algebra Summary 1
Contents
I Lecture Notes & Worked Exercises 2
1 Orthogonality and Least Squares 2
1.1 Orthogonal Projections and Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . 2
1.2 Gram–Schmidt Process and QR Factorization . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Orthogonal Transformations and Orthogonal Matrices . . . . . . . . . . . . . . . . . 4
1.4 Least Squares and Data Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Diagonalization 5
2.1 Eigenvalues, Eigenvectors, and Diagonalization . . . . . . . . . . . . . . . . . . . . . 5
2.2 Finding Eigenvalues: the Characteristic Equation . . . . . . . . . . . . . . . . . . . . 5
2.3 Algebraic and Geometric Multiplicity . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 How Eigenvalues Transform (a Useful Reference List) . . . . . . . . . . . . . . . . . 6
3 Complex Numbers and Complex Eigenvalues 7
3.1 Complex Number Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Polynomials and the Fundamental Theorem of Algebra . . . . . . . . . . . . . . . . . 7
3.3 Complex Eigenvalues of Real Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 Schur Decomposition and the Spectral Theorem 8
4.1 Real Schur Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2 Complex Inner Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 The Spectral Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.4 Geometric Classification of Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . 9
5 Quadratic Forms and Positive Definite Matrices 10
5.1 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Definiteness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Constrained Optimization of Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . 12
6 Singular Value Decomposition 12
6.1 Singular Values and the SVD Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 12
II Quick-Reference Step-by-Step Guides 14
7 Guide: Finding a Basis (Kernel, Image, Orthogonal Complement) 14
8 Guide: Orthogonal Projections, Distance, and Gram–Schmidt 14
9 Guide: Diagonalization 15
10 Guide: Complex Eigenvalues and Rotation-Scaling Matrices 15
11 Guide: Spectral Theorem, Orthogonal Diagonalization 16
12 Guide: Rotation and Reflection Matrices 16
13 Guide: Quadratic Forms 17
14 Guide: Singular Value Decomposition 17
,Mathematics 3 — Advanced Linear Algebra Summary 2
III Worked Exam Exercises 18
15 Orthogonality, Gram–Schmidt, and Least Squares 18
16 Diagonalization and Complex Eigenvalues 19
17 Spectral Theorem and Rotations/Reflections 20
18 Quadratic Forms and Positive Definite Matrices 21
19 Singular Value Decomposition 22
IV Practice Questions 23
, Mathematics 3 — Advanced Linear Algebra Summary 3
Part I Lecture Notes & Worked Ex-
ercises
1 Orthogonality and Least Squares
1.1 Orthogonal Projections and Orthonormal Bases
√
Two vectors ⃗v , w ⃗ ∈ Rn are orthogonal if ⃗v · w
⃗ = 0. The norm of ⃗v is ∥⃗v ∥ = ⃗v · ⃗v , and ⃗v is a unit
vector if ∥⃗v ∥ = 1; any nonzero ⃗v can be normalized via û = ∥⃗v1∥ ⃗v .
A vector ⃗x ∈ Rn is orthogonal to a subspace V if it is orthogonal to every vector in V .
Vectors ⃗u1 , . . . , ⃗um are orthonormal if they are all unit vectors and pairwise orthogonal: ⃗ui ·⃗uj = 1
if i = j, and 0 otherwise.
Orthonormal vectors are linearly independent, and if there are n of them in Rn , they
form a basis of Rn .
Orthogonal decomposition. For a subspace V of Rn and ⃗x ∈ Rn , there is a unique decomposi-
tion
⃗x = ⃗xq + ⃗x⊥ , ⃗xq ∈ V, ⃗x⊥ ∈ V ⊥ ,
where ⃗xq = projV (⃗x) is the orthogonal projection of ⃗x onto V . If ⃗u1 , . . . , ⃗um is an orthonormal
basis of V ,
projV (⃗x) = (⃗u1 · ⃗x)⃗u1 + · · · + (⃗um · ⃗x)⃗um .
If ⃗u1 , . . . , ⃗un is an orthonormal basis of all of Rn : ⃗x = (⃗u1 · ⃗x)⃗u1 + · · · + (⃗un · ⃗x)⃗un for every ⃗x.
The transformation T (⃗x) = projV (⃗x) is linear, and its matrix is P = QQT , where Q =
[⃗u1 · · · ⃗um ] has the orthonormal basis of V as columns; P is symmetric since P T = (QQT )T =
QQT = P .
Orthogonal complement. V ⊥ = {⃗x ∈ Rn : ⃗v · ⃗x = 0 ∀⃗v ∈ V } is the kernel of the orthogonal
projection onto V .
For a subspace V of Rn : (1) V ⊥ is a subspace; (2) V ∩ V ⊥ = {⃗0}; (3) dim(V ) + dim(V ⊥ ) = n;
(4) (V ⊥ )⊥ = V .
Key inequalities and identities.
Pythagorean theorem: ∥⃗x + ⃗y ∥2 = ∥⃗x∥2 + ∥⃗y ∥2 holds iff ⃗x ⊥ ⃗y .
Projection is a contraction: ∥projV (⃗x)∥ ≤ ∥⃗x∥, with equality iff ⃗x ∈ V .
Cauchy–Schwarz: |⃗x · ⃗y | ≤ ∥⃗x∥ ∥⃗y ∥, with equality iff ⃗x, ⃗y are parallel.
⃗x · ⃗y
Angle between vectors: θ = arccos ∈ [0, π].
∥⃗x∥∥⃗y ∥
,Mathematics 3 — Advanced Linear Algebra Summary 4
Proof of the triangle inequality via Cauchy–Schwarz (a proof pattern that recurs throughout the
course):
C-S
⃗ 2 = ∥⃗v ∥2 + ∥w∥
∥⃗v + w∥ ⃗ 2 + 2(⃗v · w)
⃗ ≤ ∥⃗v ∥2 + ∥w∥
⃗ 2 + 2∥⃗v ∥∥w∥ ⃗ 2,
⃗ = (∥⃗v ∥ + ∥w∥)
and taking square roots gives ∥⃗v + w∥
⃗ ≤ ∥⃗v ∥ + ∥w∥.
⃗
Worked example: distance to a hyperplane via the normal vector
Compute the distance from ⃗b = (1, 2, 3, 4) to the subspace V = {⃗x ∈ R4 : 4x1 −6x2 +2x3 +5x4 =
0}.
Solution. The equation defining V says ⃗n · ⃗x = 0 where ⃗n = (4, −6, 2, 5), so V ⊥ = span{⃗n}.
Since V ⊥ is one-dimensional, the projection onto V ⊥ is easy:
⃗n · ⃗b
projV ⊥ (⃗b) = ⃗n,
⃗n · ⃗n
and d(⃗b, V ) = ∥⃗b − projV (⃗b)∥ = ∥projV ⊥ (⃗b)∥ — this is much faster than projecting onto V itself
when V is a hyperplane (codimension 1).
Worked example: minimizing a linear functional on the unit sphere
Among all unit vectors ⃗u = (x, y, z) ∈ R3 , find the one minimizing x + 2y + 3z.
Solution. Write x + 2y + 3z = ⃗u · ⃗v with ⃗v = (1, 2, 3). By Cauchy–Schwarz, −∥⃗u∥∥⃗v ∥ ≤ ⃗u √ · ⃗v ≤
∥⃗u∥∥⃗v ∥, with√equality iff ⃗u, ⃗v are parallel. Since ∥⃗u∥ = 1, write ⃗u = t⃗v with 1 = ∥t⃗v√∥ = |t| 14,
so t = ±1/ 14. Since ⃗u · ⃗v = t∥⃗v ∥2 = 14t, the minimum occurs at t = −1/ 14, giving
⃗u = − √114 (1, 2, 3).
1.2 Gram–Schmidt Process and QR Factorization
Gram–Schmidt. Given a basis ⃗v1 , . . . , ⃗vm of a subspace V of Rn , resolve each ⃗vj (j = 2, . . . , m)
into components parallel and perpendicular to span(⃗v1 , . . . , ⃗vj−1 ):
1 ⊥
⃗vj⊥ = ⃗vj − (⃗u1 · ⃗vj )⃗u1 − · · · − (⃗uj−1 · ⃗vj )⃗uj−1 , ⃗uj = ⃗v ,
∥⃗vj⊥ ∥ j
with ⃗u1 = ⃗v1 /∥⃗v1 ∥. Then ⃗u1 , . . . , ⃗um is an orthonormal basis of V .
QR factorization. Any n × m matrix M with linearly independent columns ⃗v1 , . . . , ⃗vm can
be written M = QR, where Q is n × m with orthonormal columns ⃗u1 , . . . , ⃗um and R is m × m
upper triangular with positive diagonal entries. This representation is unique, with
rjj = ∥⃗vj⊥ ∥ (r11 = ∥⃗v1 ∥), rij = ⃗ui · ⃗vj for i < j, rij = 0 for i > j.
Equivalently: apply Gram–Schmidt to the columns of M to get Q; then R = QT M (since
QT Q= Im ).
Worked example: QR factorization
2 2
Find the QR factorization of M = 1 7 .
−2 −8
Solution. r11 = ∥⃗v1 ∥ = 3, ⃗u1 = 3 (2, 1, −2). Then r12 = ⃗u1 · ⃗v2 = 9, so ⃗v2⊥ = ⃗v2 − 9⃗u1 =
1
Advanced Linear Algebra — Complete Course Summary
Orthogonality & Least Squares, Diagonalization, Complex Eigenvalues,
the Spectral Theorem, Quadratic Forms, and Singular Value Decomposition
Lecture Notes, Step-by-Step Solution Guides &
Worked Exam Exercises
This document is an independent study summary and is not affiliated with, endorsed by, or a substitute for
any university’s official course materials.
,Mathematics 3 — Advanced Linear Algebra Summary 1
Contents
I Lecture Notes & Worked Exercises 2
1 Orthogonality and Least Squares 2
1.1 Orthogonal Projections and Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . 2
1.2 Gram–Schmidt Process and QR Factorization . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Orthogonal Transformations and Orthogonal Matrices . . . . . . . . . . . . . . . . . 4
1.4 Least Squares and Data Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Diagonalization 5
2.1 Eigenvalues, Eigenvectors, and Diagonalization . . . . . . . . . . . . . . . . . . . . . 5
2.2 Finding Eigenvalues: the Characteristic Equation . . . . . . . . . . . . . . . . . . . . 5
2.3 Algebraic and Geometric Multiplicity . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 How Eigenvalues Transform (a Useful Reference List) . . . . . . . . . . . . . . . . . 6
3 Complex Numbers and Complex Eigenvalues 7
3.1 Complex Number Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Polynomials and the Fundamental Theorem of Algebra . . . . . . . . . . . . . . . . . 7
3.3 Complex Eigenvalues of Real Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 Schur Decomposition and the Spectral Theorem 8
4.1 Real Schur Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2 Complex Inner Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 The Spectral Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.4 Geometric Classification of Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . 9
5 Quadratic Forms and Positive Definite Matrices 10
5.1 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Definiteness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Constrained Optimization of Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . 12
6 Singular Value Decomposition 12
6.1 Singular Values and the SVD Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 12
II Quick-Reference Step-by-Step Guides 14
7 Guide: Finding a Basis (Kernel, Image, Orthogonal Complement) 14
8 Guide: Orthogonal Projections, Distance, and Gram–Schmidt 14
9 Guide: Diagonalization 15
10 Guide: Complex Eigenvalues and Rotation-Scaling Matrices 15
11 Guide: Spectral Theorem, Orthogonal Diagonalization 16
12 Guide: Rotation and Reflection Matrices 16
13 Guide: Quadratic Forms 17
14 Guide: Singular Value Decomposition 17
,Mathematics 3 — Advanced Linear Algebra Summary 2
III Worked Exam Exercises 18
15 Orthogonality, Gram–Schmidt, and Least Squares 18
16 Diagonalization and Complex Eigenvalues 19
17 Spectral Theorem and Rotations/Reflections 20
18 Quadratic Forms and Positive Definite Matrices 21
19 Singular Value Decomposition 22
IV Practice Questions 23
, Mathematics 3 — Advanced Linear Algebra Summary 3
Part I Lecture Notes & Worked Ex-
ercises
1 Orthogonality and Least Squares
1.1 Orthogonal Projections and Orthonormal Bases
√
Two vectors ⃗v , w ⃗ ∈ Rn are orthogonal if ⃗v · w
⃗ = 0. The norm of ⃗v is ∥⃗v ∥ = ⃗v · ⃗v , and ⃗v is a unit
vector if ∥⃗v ∥ = 1; any nonzero ⃗v can be normalized via û = ∥⃗v1∥ ⃗v .
A vector ⃗x ∈ Rn is orthogonal to a subspace V if it is orthogonal to every vector in V .
Vectors ⃗u1 , . . . , ⃗um are orthonormal if they are all unit vectors and pairwise orthogonal: ⃗ui ·⃗uj = 1
if i = j, and 0 otherwise.
Orthonormal vectors are linearly independent, and if there are n of them in Rn , they
form a basis of Rn .
Orthogonal decomposition. For a subspace V of Rn and ⃗x ∈ Rn , there is a unique decomposi-
tion
⃗x = ⃗xq + ⃗x⊥ , ⃗xq ∈ V, ⃗x⊥ ∈ V ⊥ ,
where ⃗xq = projV (⃗x) is the orthogonal projection of ⃗x onto V . If ⃗u1 , . . . , ⃗um is an orthonormal
basis of V ,
projV (⃗x) = (⃗u1 · ⃗x)⃗u1 + · · · + (⃗um · ⃗x)⃗um .
If ⃗u1 , . . . , ⃗un is an orthonormal basis of all of Rn : ⃗x = (⃗u1 · ⃗x)⃗u1 + · · · + (⃗un · ⃗x)⃗un for every ⃗x.
The transformation T (⃗x) = projV (⃗x) is linear, and its matrix is P = QQT , where Q =
[⃗u1 · · · ⃗um ] has the orthonormal basis of V as columns; P is symmetric since P T = (QQT )T =
QQT = P .
Orthogonal complement. V ⊥ = {⃗x ∈ Rn : ⃗v · ⃗x = 0 ∀⃗v ∈ V } is the kernel of the orthogonal
projection onto V .
For a subspace V of Rn : (1) V ⊥ is a subspace; (2) V ∩ V ⊥ = {⃗0}; (3) dim(V ) + dim(V ⊥ ) = n;
(4) (V ⊥ )⊥ = V .
Key inequalities and identities.
Pythagorean theorem: ∥⃗x + ⃗y ∥2 = ∥⃗x∥2 + ∥⃗y ∥2 holds iff ⃗x ⊥ ⃗y .
Projection is a contraction: ∥projV (⃗x)∥ ≤ ∥⃗x∥, with equality iff ⃗x ∈ V .
Cauchy–Schwarz: |⃗x · ⃗y | ≤ ∥⃗x∥ ∥⃗y ∥, with equality iff ⃗x, ⃗y are parallel.
⃗x · ⃗y
Angle between vectors: θ = arccos ∈ [0, π].
∥⃗x∥∥⃗y ∥
,Mathematics 3 — Advanced Linear Algebra Summary 4
Proof of the triangle inequality via Cauchy–Schwarz (a proof pattern that recurs throughout the
course):
C-S
⃗ 2 = ∥⃗v ∥2 + ∥w∥
∥⃗v + w∥ ⃗ 2 + 2(⃗v · w)
⃗ ≤ ∥⃗v ∥2 + ∥w∥
⃗ 2 + 2∥⃗v ∥∥w∥ ⃗ 2,
⃗ = (∥⃗v ∥ + ∥w∥)
and taking square roots gives ∥⃗v + w∥
⃗ ≤ ∥⃗v ∥ + ∥w∥.
⃗
Worked example: distance to a hyperplane via the normal vector
Compute the distance from ⃗b = (1, 2, 3, 4) to the subspace V = {⃗x ∈ R4 : 4x1 −6x2 +2x3 +5x4 =
0}.
Solution. The equation defining V says ⃗n · ⃗x = 0 where ⃗n = (4, −6, 2, 5), so V ⊥ = span{⃗n}.
Since V ⊥ is one-dimensional, the projection onto V ⊥ is easy:
⃗n · ⃗b
projV ⊥ (⃗b) = ⃗n,
⃗n · ⃗n
and d(⃗b, V ) = ∥⃗b − projV (⃗b)∥ = ∥projV ⊥ (⃗b)∥ — this is much faster than projecting onto V itself
when V is a hyperplane (codimension 1).
Worked example: minimizing a linear functional on the unit sphere
Among all unit vectors ⃗u = (x, y, z) ∈ R3 , find the one minimizing x + 2y + 3z.
Solution. Write x + 2y + 3z = ⃗u · ⃗v with ⃗v = (1, 2, 3). By Cauchy–Schwarz, −∥⃗u∥∥⃗v ∥ ≤ ⃗u √ · ⃗v ≤
∥⃗u∥∥⃗v ∥, with√equality iff ⃗u, ⃗v are parallel. Since ∥⃗u∥ = 1, write ⃗u = t⃗v with 1 = ∥t⃗v√∥ = |t| 14,
so t = ±1/ 14. Since ⃗u · ⃗v = t∥⃗v ∥2 = 14t, the minimum occurs at t = −1/ 14, giving
⃗u = − √114 (1, 2, 3).
1.2 Gram–Schmidt Process and QR Factorization
Gram–Schmidt. Given a basis ⃗v1 , . . . , ⃗vm of a subspace V of Rn , resolve each ⃗vj (j = 2, . . . , m)
into components parallel and perpendicular to span(⃗v1 , . . . , ⃗vj−1 ):
1 ⊥
⃗vj⊥ = ⃗vj − (⃗u1 · ⃗vj )⃗u1 − · · · − (⃗uj−1 · ⃗vj )⃗uj−1 , ⃗uj = ⃗v ,
∥⃗vj⊥ ∥ j
with ⃗u1 = ⃗v1 /∥⃗v1 ∥. Then ⃗u1 , . . . , ⃗um is an orthonormal basis of V .
QR factorization. Any n × m matrix M with linearly independent columns ⃗v1 , . . . , ⃗vm can
be written M = QR, where Q is n × m with orthonormal columns ⃗u1 , . . . , ⃗um and R is m × m
upper triangular with positive diagonal entries. This representation is unique, with
rjj = ∥⃗vj⊥ ∥ (r11 = ∥⃗v1 ∥), rij = ⃗ui · ⃗vj for i < j, rij = 0 for i > j.
Equivalently: apply Gram–Schmidt to the columns of M to get Q; then R = QT M (since
QT Q= Im ).
Worked example: QR factorization
2 2
Find the QR factorization of M = 1 7 .
−2 −8
Solution. r11 = ∥⃗v1 ∥ = 3, ⃗u1 = 3 (2, 1, −2). Then r12 = ⃗u1 · ⃗v2 = 9, so ⃗v2⊥ = ⃗v2 − 9⃗u1 =
1
