SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linear Algebra and Optimization: An Introduction 1
2 Linear Transformations and Linear Systems 17
3 Diagonalizable Matrices and Eigenvectors 35
4 Optimization Basics: A Machine Learning View 47
5 Optimization Challenges and Advanced Solutions 57
6 Lagrangian Relaxation and Duality 63
7 Singular Value Decomposition 71
8 Matrix Factorization 81
9 The Linear Algebra of Similarity 89
10 The Linear Algebra of Graphs 95
11 Optimization in Computational Graphs 101
viii
,Chapter j 1
Linear jAlgebra jand jOptimization: jAn jIntroduction
1. For j any j two j vectors j x j and j y, j which j are j each j of j length j a, j show j that j (i)
j x j− j y j is jorthogonal jto jx j+ jy, j and j(ii) j the jdot jproduct jof jx j− j3y j and jx j+ j3y j is
j negative.
(i) jThe jfirst jis jsimply · −jx· j x j y j y jusing jthe jdistributive jproperty jof jmatrix
j multiplication. j The j dot j product j of j a j vector j with jitself j is jits j squared jlength. j Since
jboth j vectors j are j of j the j same jlength, jit j follows j that j the j result jis j 0. j (ii) j In jthe j second
jcase, j one j can j use j a j similar j argument j to j show j that j the j result jis j a j − j 9a , j which jis
2 2
jnegative.
2. Consider j a j situation j in j which j you j have j three j matrices j A, j B, j and j C, j of j sizes
j 10 j× j2, j2 j× j10, jand j 10 j× j10, j respectively.
(a) Suppose jyou jhad jto jcompute jthe jmatrix jproduct jABC. jFrom jan jefficiency
jper- jspective, jwould jit jcomputationally jmake jmore jsense jto jcompute j(AB)C
jor jwould jit jmake jmore jsense jto jcompute jA(BC)?
(b) If jyou jhad jto jcompute jthe jmatrix jproduct jCAB, jwould jit jmake jmore jsense
jto jcompute j (CA)B j or j C(AB)?
The jmain jpoint jis jto jkeep jthe jsize jof jthe jintermediate jmatrix jas jsmall jas
jpossible j in jorder jto jreduce jboth jcomputational jand jspace jrequirements. jIn jthe
jcase jof jABC, jit jmakes jsense jto jcompute jBC jfirst. jIn jthe jcase jof jCAB jit jmakes
jsense jto jcompute jCA jfirst. jThis jtype jof jassociativity jproperty jis jused jfrequently
jin jmachine jlearning jin jorder jto jreduce jcomputational jrequirements.
3. Show j that j if j a j matrix j A j satisfies j—A j = ATj, j then j all j the j diagonal
j elements j of j the jmatrix jare j0.
Note jthat jA j+ jAT j= j0. jHowever, jthis jmatrix jalso jcontains jtwice jthe jdiagonal
jelements jof jA jon jits jdiagonal. jTherefore, jthe jdiagonal jelements jof jA jmust jbe
j0.
4. Show jthat jif jwe jhave ja jmatrix jsatisfying jA—j= ATj, jthen jfor jany jcolumn jvector jx,
jwe jhave j x j Ax j= j0.
T
Note j that j the j transpose j of j the j scalar j xT jAx j remains j unchanged. j Therefore, j we
1
, j have
xTjAx j= j(xTjAx)T j = jxTjATjx j= j−xT jAx. j Therefore, j we j have j 2xTjAx j= j0.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linear Algebra and Optimization: An Introduction 1
2 Linear Transformations and Linear Systems 17
3 Diagonalizable Matrices and Eigenvectors 35
4 Optimization Basics: A Machine Learning View 47
5 Optimization Challenges and Advanced Solutions 57
6 Lagrangian Relaxation and Duality 63
7 Singular Value Decomposition 71
8 Matrix Factorization 81
9 The Linear Algebra of Similarity 89
10 The Linear Algebra of Graphs 95
11 Optimization in Computational Graphs 101
viii
,Chapter j 1
Linear jAlgebra jand jOptimization: jAn jIntroduction
1. For j any j two j vectors j x j and j y, j which j are j each j of j length j a, j show j that j (i)
j x j− j y j is jorthogonal jto jx j+ jy, j and j(ii) j the jdot jproduct jof jx j− j3y j and jx j+ j3y j is
j negative.
(i) jThe jfirst jis jsimply · −jx· j x j y j y jusing jthe jdistributive jproperty jof jmatrix
j multiplication. j The j dot j product j of j a j vector j with jitself j is jits j squared jlength. j Since
jboth j vectors j are j of j the j same jlength, jit j follows j that j the j result jis j 0. j (ii) j In jthe j second
jcase, j one j can j use j a j similar j argument j to j show j that j the j result jis j a j − j 9a , j which jis
2 2
jnegative.
2. Consider j a j situation j in j which j you j have j three j matrices j A, j B, j and j C, j of j sizes
j 10 j× j2, j2 j× j10, jand j 10 j× j10, j respectively.
(a) Suppose jyou jhad jto jcompute jthe jmatrix jproduct jABC. jFrom jan jefficiency
jper- jspective, jwould jit jcomputationally jmake jmore jsense jto jcompute j(AB)C
jor jwould jit jmake jmore jsense jto jcompute jA(BC)?
(b) If jyou jhad jto jcompute jthe jmatrix jproduct jCAB, jwould jit jmake jmore jsense
jto jcompute j (CA)B j or j C(AB)?
The jmain jpoint jis jto jkeep jthe jsize jof jthe jintermediate jmatrix jas jsmall jas
jpossible j in jorder jto jreduce jboth jcomputational jand jspace jrequirements. jIn jthe
jcase jof jABC, jit jmakes jsense jto jcompute jBC jfirst. jIn jthe jcase jof jCAB jit jmakes
jsense jto jcompute jCA jfirst. jThis jtype jof jassociativity jproperty jis jused jfrequently
jin jmachine jlearning jin jorder jto jreduce jcomputational jrequirements.
3. Show j that j if j a j matrix j A j satisfies j—A j = ATj, j then j all j the j diagonal
j elements j of j the jmatrix jare j0.
Note jthat jA j+ jAT j= j0. jHowever, jthis jmatrix jalso jcontains jtwice jthe jdiagonal
jelements jof jA jon jits jdiagonal. jTherefore, jthe jdiagonal jelements jof jA jmust jbe
j0.
4. Show jthat jif jwe jhave ja jmatrix jsatisfying jA—j= ATj, jthen jfor jany jcolumn jvector jx,
jwe jhave j x j Ax j= j0.
T
Note j that j the j transpose j of j the j scalar j xT jAx j remains j unchanged. j Therefore, j we
1
, j have
xTjAx j= j(xTjAx)T j = jxTjATjx j= j−xT jAx. j Therefore, j we j have j 2xTjAx j= j0.
2
