SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearz Algebraz andz Optimization:z Anz Introduction 1
2 Linearz Transformationsz andz Linearz Systems 17
3 Diagonalizablez Matricesz andz Eigenvectors 35
4 OptimizationzBasics:zAzMachinezLearningzView 47
5 Optimizationz Challengesz andz Advancedz Solutions 57
6 Lagrangianz Relaxationz andz Duality 63
7 Singularz Valuez Decomposition 71
8 Matrixz Factorization 81
9 Thez Linearz Algebraz ofz Similarity 89
10 Thez Linearz Algebraz ofz Graphs 95
11 Optimizationz inz Computationalz Graphs 101
viii
,Chapterz 1
LinearzAlgebrazandzOptimization:zAnzIntroduction
1. Forz anyz twoz vectorsz xz andz y,z whichz arez eachz ofz lengthz a,z showz thatz (i)z xz
−zyz iszorthogonalztozxz+zy,z andz(ii)z thezdotzproductzofzxz−z3yz andzxz+z3yz isz ne
gative.
(i)zThezfirstziszsimply xzz x· z yz yzusingzthezdistributivezpropertyzofzmatrixzmu
·z z−
ltiplication.zThezdotzproductzofzazvectorzwithzitselfziszitszsquaredzlength.zSi
ncezbothzvectorszarezofzthezsamezlength,zitzfollowszthatzthezresultzisz0.z(ii)zI
nzthezsecondzcase,zonezcanzusezazsimilarzargumentztozshowzthatzthezresultzis
za 2z− z9a2,zwhichzisznegative.
2. Considerz az situationz inz whichz youz havez threez matricesz A,z B,z andz C,z ofz sizesz 1
0z×z2,z2z×z10,zandz 10z×z10,z respectively.
(a) SupposezyouzhadztozcomputezthezmatrixzproductzABC.zFromzanzefficiencyz
per-
zspective,zwouldzitzcomputationallyzmakezmorezsenseztozcomputez(AB)Czorzw
ouldzitzmakezmorezsenseztozcomputezA(BC)?
(b) IfzyouzhadztozcomputezthezmatrixzproductzCAB,zwouldzitzmakezmorezsensez
tozcomputez (CA)Bz orz C(AB)?
Thezmainzpointzisztozkeepzthezsizezofzthezintermediatezmatrixzaszsmallza
szpossiblez inzorderztozreducezbothzcomputationalzandzspacezrequiremen
ts.zInzthezcasezofzABC,zitzmakeszsenseztozcomputezBCzfirst.zInzthezcasezofz
CABzitzmakeszsenseztozcomputezCAzfirst.zThisztypezofzassociativityzprop
ertyziszusedzfrequentlyzinzmachinezlearningzinzorderztozreducezcomputat
ionalzrequirements.
3. Showz thatz ifz az matrixz Az satisfiesz A—z =
ATz,z thenz allz thez diagonalz elementsz ofz t
hezmatrixzarez0.
NotezthatzAz+zATz=z0.zHowever,zthiszmatrixzalsozcontainsztwicezthezdiag
onalzelementszofzAzonzitszdiagonal.zTherefore,zthezdiagonalzelementszofz
Azmustzbez0.
4. Showzthatzifzwezhavezazmatrixzsatisfying—zAz=
1
, ATz,zthenzforzanyzcolumnzvectorzx,z
wezhavez x zAxz=z0.
T
Notez thatz thez transposez ofz thez scalarz xTzAxz remainsz unchanged.z Therefore,z wez
have
xTzAxz=z(xTzAx)Tz =zxTzATzxz=z−xTzAx.z Therefore,z wez havez 2xTzAxz=z0.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearz Algebraz andz Optimization:z Anz Introduction 1
2 Linearz Transformationsz andz Linearz Systems 17
3 Diagonalizablez Matricesz andz Eigenvectors 35
4 OptimizationzBasics:zAzMachinezLearningzView 47
5 Optimizationz Challengesz andz Advancedz Solutions 57
6 Lagrangianz Relaxationz andz Duality 63
7 Singularz Valuez Decomposition 71
8 Matrixz Factorization 81
9 Thez Linearz Algebraz ofz Similarity 89
10 Thez Linearz Algebraz ofz Graphs 95
11 Optimizationz inz Computationalz Graphs 101
viii
,Chapterz 1
LinearzAlgebrazandzOptimization:zAnzIntroduction
1. Forz anyz twoz vectorsz xz andz y,z whichz arez eachz ofz lengthz a,z showz thatz (i)z xz
−zyz iszorthogonalztozxz+zy,z andz(ii)z thezdotzproductzofzxz−z3yz andzxz+z3yz isz ne
gative.
(i)zThezfirstziszsimply xzz x· z yz yzusingzthezdistributivezpropertyzofzmatrixzmu
·z z−
ltiplication.zThezdotzproductzofzazvectorzwithzitselfziszitszsquaredzlength.zSi
ncezbothzvectorszarezofzthezsamezlength,zitzfollowszthatzthezresultzisz0.z(ii)zI
nzthezsecondzcase,zonezcanzusezazsimilarzargumentztozshowzthatzthezresultzis
za 2z− z9a2,zwhichzisznegative.
2. Considerz az situationz inz whichz youz havez threez matricesz A,z B,z andz C,z ofz sizesz 1
0z×z2,z2z×z10,zandz 10z×z10,z respectively.
(a) SupposezyouzhadztozcomputezthezmatrixzproductzABC.zFromzanzefficiencyz
per-
zspective,zwouldzitzcomputationallyzmakezmorezsenseztozcomputez(AB)Czorzw
ouldzitzmakezmorezsenseztozcomputezA(BC)?
(b) IfzyouzhadztozcomputezthezmatrixzproductzCAB,zwouldzitzmakezmorezsensez
tozcomputez (CA)Bz orz C(AB)?
Thezmainzpointzisztozkeepzthezsizezofzthezintermediatezmatrixzaszsmallza
szpossiblez inzorderztozreducezbothzcomputationalzandzspacezrequiremen
ts.zInzthezcasezofzABC,zitzmakeszsenseztozcomputezBCzfirst.zInzthezcasezofz
CABzitzmakeszsenseztozcomputezCAzfirst.zThisztypezofzassociativityzprop
ertyziszusedzfrequentlyzinzmachinezlearningzinzorderztozreducezcomputat
ionalzrequirements.
3. Showz thatz ifz az matrixz Az satisfiesz A—z =
ATz,z thenz allz thez diagonalz elementsz ofz t
hezmatrixzarez0.
NotezthatzAz+zATz=z0.zHowever,zthiszmatrixzalsozcontainsztwicezthezdiag
onalzelementszofzAzonzitszdiagonal.zTherefore,zthezdiagonalzelementszofz
Azmustzbez0.
4. Showzthatzifzwezhavezazmatrixzsatisfying—zAz=
1
, ATz,zthenzforzanyzcolumnzvectorzx,z
wezhavez x zAxz=z0.
T
Notez thatz thez transposez ofz thez scalarz xTzAxz remainsz unchanged.z Therefore,z wez
have
xTzAxz=z(xTzAx)Tz =zxTzATzxz=z−xTzAx.z Therefore,z wez havez 2xTzAxz=z0.
2
