SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearm Algebram andm Optimization:m Anm Introduction 1
2 Linearm Transformations m andm Linearm Systems 17
3 Diagonalizable m Matricesm andm Eigenvectors 35
4 OptimizationmBasics:mAmMachinemLearningmView 47
5 Optimization m Challengesm andm Advancedm Solutions 57
6 Lagrangianm Relaxationm andm Duality 63
7 Singularm Valuem Decomposition 71
8 Matrixm Factorization 81
9 Them Linearm Algebram ofm Similarity 89
10 Them Linearm Algebram ofm Graphs 95
11 Optimizationm inm Computationalm Graphs 101
viii
,Chapterm 1
LinearmAlgebramandmOptimization:mAnmIntroduction
1. Form anym twom vectorsm xm andm y,m whichm arem eachm ofm lengthm a,m showm thatm (i
)m xm−mym ismorthogonalmtomxm+my,m andm(ii)m themdotmproductmofmxm−m3ym andmxm
+m3ym ism negative.
(i)mThemfirstmismsimply
·m −mmx·m xm ym ymusingmthemdistributivempropertymofmmatri
xmmultiplication.mThemdotmproductmofmamvectormwithmitselfmismitsmsquaredm
length.mSincembothmvectorsmaremofmthemsamemlength,mitmfollowsmthatmthemr
esultmism0.m(ii)mInmthemsecondmcase,monemcanmusemamsimilarmargumentmtomsh
owmthatmthemresultmisma2m−m9a2,mwhichmismnegative.
2. Considerm am situationm inm whichm youm havem threem matricesm A,m B,m andm C,m ofm si
zesm 10m×m2,m2m×m10,mandm10m×m10,mrespectively.
(a) SupposemyoumhadmtomcomputemthemmatrixmproductmABC.mFrommanmefficie
ncymper-
mspective,mwouldmitmcomputationallymmakemmoremsensemtomcomputem(AB)Cm
ormwouldmitmmakemmoremsensemtomcomputemA(BC)?
(b) IfmyoumhadmtomcomputemthemmatrixmproductmCAB,mwouldmitmmakemmorems
ensemtomcomputem (CA)Bm orm C(AB)?
Themmainmpointmismtomkeepmthemsizemofmthemintermediatemmatrixmasms
mallmasmpossiblem inmordermtomreducembothmcomputationalmandmspacemr
equirements.mInmthemcasemofmABC,mitmmakesmsensemtomcomputemBCmfirs
t.mInmthemcasemofmCABmitmmakesmsensemtomcomputemCAmfirst.mThismtype
mofmassociativity mpropertymis mused mfrequentlyminmmachinemlearningminm
ordermtomreducemcomputationalmrequirements.
3. Showm thatm ifm am matrixm Am satisfies—m Am =
ATm,m thenm allm them diagonalm elementsm o
fmthemmatrixmarem0.
NotemthatmAm+mATm=m0.mHowever,mthismmatrixmalsomcontainsmtwicemthe
mdiagonalmelementsmofm A mon mitsmdiagonal.mTherefore,mthemdiagonalmele
mentsmofmAmmustmbem0.
4. Showmthatmifmwemhavemammatrixmsatisfying
— mAm=
1
, ATm,mthenmformanymcolumnmvectormx
,mwemhavem x mAxm=m0.
T
Notem thatm them transposem ofm them scalarm xTmAxm remainsm unchanged.m Therefore,
m wem have
xTmAxm=m(xTmAx)Tm =mxTmATmxm=m−xTmAx.m Therefore,m wem havem 2xTmAxm=
m0.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearm Algebram andm Optimization:m Anm Introduction 1
2 Linearm Transformations m andm Linearm Systems 17
3 Diagonalizable m Matricesm andm Eigenvectors 35
4 OptimizationmBasics:mAmMachinemLearningmView 47
5 Optimization m Challengesm andm Advancedm Solutions 57
6 Lagrangianm Relaxationm andm Duality 63
7 Singularm Valuem Decomposition 71
8 Matrixm Factorization 81
9 Them Linearm Algebram ofm Similarity 89
10 Them Linearm Algebram ofm Graphs 95
11 Optimizationm inm Computationalm Graphs 101
viii
,Chapterm 1
LinearmAlgebramandmOptimization:mAnmIntroduction
1. Form anym twom vectorsm xm andm y,m whichm arem eachm ofm lengthm a,m showm thatm (i
)m xm−mym ismorthogonalmtomxm+my,m andm(ii)m themdotmproductmofmxm−m3ym andmxm
+m3ym ism negative.
(i)mThemfirstmismsimply
·m −mmx·m xm ym ymusingmthemdistributivempropertymofmmatri
xmmultiplication.mThemdotmproductmofmamvectormwithmitselfmismitsmsquaredm
length.mSincembothmvectorsmaremofmthemsamemlength,mitmfollowsmthatmthemr
esultmism0.m(ii)mInmthemsecondmcase,monemcanmusemamsimilarmargumentmtomsh
owmthatmthemresultmisma2m−m9a2,mwhichmismnegative.
2. Considerm am situationm inm whichm youm havem threem matricesm A,m B,m andm C,m ofm si
zesm 10m×m2,m2m×m10,mandm10m×m10,mrespectively.
(a) SupposemyoumhadmtomcomputemthemmatrixmproductmABC.mFrommanmefficie
ncymper-
mspective,mwouldmitmcomputationallymmakemmoremsensemtomcomputem(AB)Cm
ormwouldmitmmakemmoremsensemtomcomputemA(BC)?
(b) IfmyoumhadmtomcomputemthemmatrixmproductmCAB,mwouldmitmmakemmorems
ensemtomcomputem (CA)Bm orm C(AB)?
Themmainmpointmismtomkeepmthemsizemofmthemintermediatemmatrixmasms
mallmasmpossiblem inmordermtomreducembothmcomputationalmandmspacemr
equirements.mInmthemcasemofmABC,mitmmakesmsensemtomcomputemBCmfirs
t.mInmthemcasemofmCABmitmmakesmsensemtomcomputemCAmfirst.mThismtype
mofmassociativity mpropertymis mused mfrequentlyminmmachinemlearningminm
ordermtomreducemcomputationalmrequirements.
3. Showm thatm ifm am matrixm Am satisfies—m Am =
ATm,m thenm allm them diagonalm elementsm o
fmthemmatrixmarem0.
NotemthatmAm+mATm=m0.mHowever,mthismmatrixmalsomcontainsmtwicemthe
mdiagonalmelementsmofm A mon mitsmdiagonal.mTherefore,mthemdiagonalmele
mentsmofmAmmustmbem0.
4. Showmthatmifmwemhavemammatrixmsatisfying
— mAm=
1
, ATm,mthenmformanymcolumnmvectormx
,mwemhavem x mAxm=m0.
T
Notem thatm them transposem ofm them scalarm xTmAxm remainsm unchanged.m Therefore,
m wem have
xTmAxm=m(xTmAx)Tm =mxTmATmxm=m−xTmAx.m Therefore,m wem havem 2xTmAxm=
m0.
2
