LINEAR ALGEBRA AND OPTIMIZATION
FOR MACHINE LEARNING
COMPREHENSIVE EXAM SCRIPT VERIFIED
QUESTIONS AND SOLUTIONS GRADED
APLUS
●● What does the gradient of the least squares objective equal?
Answer: $ \nabla f(x) = 2A^\top(Ax-b) $.
●● What is the geometric meaning of least squares?
Answer: Least squares chooses $ \hat x $ so that $ A\hat x $ is the
orthogonal projection of $ b $ onto the column space of $ A $.
●● What is the condition for the uniqueness of the least-squares
solution?
Answer: The solution is unique if $ A^\top A $ is invertible (i.e.,
columns of $ A $ are linearly independent).
●● What is the ridge regression objective?
Answer: $ \min_x \|Ax-b\|_2^2 + \lambda\|x\|_2^2 $.
●● What are the normal equations for ridge regression?
, Answer: $ (A^\top A + \lambda I)x = A^\top b $.
●● What is the solution to the ridge regression problem?
Answer: $ x = (A^\top A + \lambda I)^{-1}A^\top b $.
●● What is the general Tikhonov regularization objective?
Answer: $ \min_x \|Ax-b\|_2^2 + \lambda\|Wx\|_2^2 $.
●● What are the normal equations for general Tikhonov regularization?
Answer: $ (A^\top A + \lambda W^\top W)x = A^\top b $.
●● How do you convert pairwise differences into a matrix equation?
Answer: If a measurement says $ x_i - x_j \approx d $, create one row in
$ A $ with $ +1 $ in column $ i $, $ -1 $ in column $ j $, and set the
corresponding entry of $ b $ to $ d $.
●● What does the denoising regularizer using a difference matrix
penalize?
Answer: It penalizes roughness by enforcing $ \|Dx\|_2^2 =
\sum_{k=1}^{n-1}(x_{k+1}-x_k)^2 $.
●● What is the test for a symmetric matrix to be positive definite?
FOR MACHINE LEARNING
COMPREHENSIVE EXAM SCRIPT VERIFIED
QUESTIONS AND SOLUTIONS GRADED
APLUS
●● What does the gradient of the least squares objective equal?
Answer: $ \nabla f(x) = 2A^\top(Ax-b) $.
●● What is the geometric meaning of least squares?
Answer: Least squares chooses $ \hat x $ so that $ A\hat x $ is the
orthogonal projection of $ b $ onto the column space of $ A $.
●● What is the condition for the uniqueness of the least-squares
solution?
Answer: The solution is unique if $ A^\top A $ is invertible (i.e.,
columns of $ A $ are linearly independent).
●● What is the ridge regression objective?
Answer: $ \min_x \|Ax-b\|_2^2 + \lambda\|x\|_2^2 $.
●● What are the normal equations for ridge regression?
, Answer: $ (A^\top A + \lambda I)x = A^\top b $.
●● What is the solution to the ridge regression problem?
Answer: $ x = (A^\top A + \lambda I)^{-1}A^\top b $.
●● What is the general Tikhonov regularization objective?
Answer: $ \min_x \|Ax-b\|_2^2 + \lambda\|Wx\|_2^2 $.
●● What are the normal equations for general Tikhonov regularization?
Answer: $ (A^\top A + \lambda W^\top W)x = A^\top b $.
●● How do you convert pairwise differences into a matrix equation?
Answer: If a measurement says $ x_i - x_j \approx d $, create one row in
$ A $ with $ +1 $ in column $ i $, $ -1 $ in column $ j $, and set the
corresponding entry of $ b $ to $ d $.
●● What does the denoising regularizer using a difference matrix
penalize?
Answer: It penalizes roughness by enforcing $ \|Dx\|_2^2 =
\sum_{k=1}^{n-1}(x_{k+1}-x_k)^2 $.
●● What is the test for a symmetric matrix to be positive definite?
