SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearl Algebral andl Optimization:l Anl Introduction 1
2 Linearl Transformationsl andl Linearl Systems 17
3 Diagonalizablel Matricesl andl Eigenvectors 35
4 OptimizationlBasics:lAlMachinelLearninglView 47
5 Optimizationl Challengesl andl Advancedl Solutions 57
6 Lagrangianl Relaxationl andl Duality 63
7 Singularl Valuel Decomposition 71
8 Matrixl Factorization 81
9 Thel Linearl Algebral ofl Similarity 89
10 Thel Linearl Algebral ofl Graphs 95
11 Optimizationl inl Computationall Graphs 101
viii
,Chapterl 1
LinearlAlgebralandlOptimization:lAnlIntroduction
1. Forl anyl twol vectorsl xl andl y,l whichl arel eachl ofl lengthl a,l showl thatl (i)l xl−lyl isl
orthogonalltolxl+ly,l andl(ii)l theldotlproductloflxl−l3yl andlxl+l3yl isl negative.
(i)lThelfirstlislsimply·llx−l xl ·l yl ylusingltheldistributivelpropertyloflmatrixlmultip
lication.lTheldotlproductloflalvectorlwithlitselflislitslsquaredllength.lSincelbo
thlvectorslareloflthelsamellength,litlfollowslthatlthelresultlisl0.l(ii)lInlthelsecon
dlcase,lonelcanluselalsimilarlargumentltolshowlthatlthelresultlisla2l−l9a2,lwhic
hlislnegative.
2. Considerl al situationl inl whichl youl havel threel matricesl A,l B,l andl C,l ofl sizesl 10l×l
2,l2l×l10,landl10l×l10,lrespectively.
(a) SupposelyoulhadltolcomputelthelmatrixlproductlABC.lFromlanlefficiencylpe
r-
lspective,lwouldlitlcomputationallylmakelmorelsenseltolcomputel(AB)Clorlwoul
dlitlmakelmorelsenseltolcomputelA(BC)?
(b) IflyoulhadltolcomputelthelmatrixlproductlCAB,lwouldlitlmakelmorelsenseltolc
omputel (CA)Bl orl C(AB)?
Thelmainlpointlisltolkeeplthelsizeloflthelintermediatelmatrixlaslsmalllaslpo
ssiblel inlorderltolreducelbothlcomputationallandlspacelrequirements.lInlt
helcaseloflABC,litlmakeslsenseltolcomputelBClfirst.lInlthelcaseloflCABlitlma
keslsenseltolcomputelCAlfirst.lThisltypeloflassociativitylpropertylislusedlfr
equentlylinlmachinellearninglinlorderltolreducelcomputationallrequirem
ents.
3. —
Showl thatl ifl al matrixl Al satisfiesl Al =
ATl,l thenl alll thel diagonall elementsl ofl thel
matrixlarel0.
NotelthatlAl+lATl=l0.lHowever,lthislmatrixlalsolcontainsltwiceltheldiagona
llelementsloflAlonlitsldiagonal.lTherefore,ltheldiagonallelementsloflAlmus
tlbel0.
4. ShowlthatliflwelhavelalmatrixlsatisfyinglA—l=
ATl,lthenlforlanylcolumnlvectorlx,lwel
1
, havel xTlAxl=l0.
Notel thatl thel transposel ofl thel scalarl xTlAxl remainsl unchanged.l Therefore,l wel ha
ve
xTlAxl=l(xTlAx)Tl =lxTlATlxl=l−xTlAx.l Therefore,l wel havel 2xTlAxl=l0.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearl Algebral andl Optimization:l Anl Introduction 1
2 Linearl Transformationsl andl Linearl Systems 17
3 Diagonalizablel Matricesl andl Eigenvectors 35
4 OptimizationlBasics:lAlMachinelLearninglView 47
5 Optimizationl Challengesl andl Advancedl Solutions 57
6 Lagrangianl Relaxationl andl Duality 63
7 Singularl Valuel Decomposition 71
8 Matrixl Factorization 81
9 Thel Linearl Algebral ofl Similarity 89
10 Thel Linearl Algebral ofl Graphs 95
11 Optimizationl inl Computationall Graphs 101
viii
,Chapterl 1
LinearlAlgebralandlOptimization:lAnlIntroduction
1. Forl anyl twol vectorsl xl andl y,l whichl arel eachl ofl lengthl a,l showl thatl (i)l xl−lyl isl
orthogonalltolxl+ly,l andl(ii)l theldotlproductloflxl−l3yl andlxl+l3yl isl negative.
(i)lThelfirstlislsimply·llx−l xl ·l yl ylusingltheldistributivelpropertyloflmatrixlmultip
lication.lTheldotlproductloflalvectorlwithlitselflislitslsquaredllength.lSincelbo
thlvectorslareloflthelsamellength,litlfollowslthatlthelresultlisl0.l(ii)lInlthelsecon
dlcase,lonelcanluselalsimilarlargumentltolshowlthatlthelresultlisla2l−l9a2,lwhic
hlislnegative.
2. Considerl al situationl inl whichl youl havel threel matricesl A,l B,l andl C,l ofl sizesl 10l×l
2,l2l×l10,landl10l×l10,lrespectively.
(a) SupposelyoulhadltolcomputelthelmatrixlproductlABC.lFromlanlefficiencylpe
r-
lspective,lwouldlitlcomputationallylmakelmorelsenseltolcomputel(AB)Clorlwoul
dlitlmakelmorelsenseltolcomputelA(BC)?
(b) IflyoulhadltolcomputelthelmatrixlproductlCAB,lwouldlitlmakelmorelsenseltolc
omputel (CA)Bl orl C(AB)?
Thelmainlpointlisltolkeeplthelsizeloflthelintermediatelmatrixlaslsmalllaslpo
ssiblel inlorderltolreducelbothlcomputationallandlspacelrequirements.lInlt
helcaseloflABC,litlmakeslsenseltolcomputelBClfirst.lInlthelcaseloflCABlitlma
keslsenseltolcomputelCAlfirst.lThisltypeloflassociativitylpropertylislusedlfr
equentlylinlmachinellearninglinlorderltolreducelcomputationallrequirem
ents.
3. —
Showl thatl ifl al matrixl Al satisfiesl Al =
ATl,l thenl alll thel diagonall elementsl ofl thel
matrixlarel0.
NotelthatlAl+lATl=l0.lHowever,lthislmatrixlalsolcontainsltwiceltheldiagona
llelementsloflAlonlitsldiagonal.lTherefore,ltheldiagonallelementsloflAlmus
tlbel0.
4. ShowlthatliflwelhavelalmatrixlsatisfyinglA—l=
ATl,lthenlforlanylcolumnlvectorlx,lwel
1
, havel xTlAxl=l0.
Notel thatl thel transposel ofl thel scalarl xTlAxl remainsl unchanged.l Therefore,l wel ha
ve
xTlAxl=l(xTlAx)Tl =lxTlATlxl=l−xTlAx.l Therefore,l wel havel 2xTlAxl=l0.
2
