SOLUTIONMANUAL
Linear Algebra and Optimization for Machine Learning
1st Edition by Charu Aggarwal. Chapters 1–11
vii
,Contents
1 LinearR AlgebraR andR Optimization:R AnR Introduction 1
2 LinearR Transformations R andR LinearR Systems 17
3 Diagonalizable R MatricesR andR Eigenvectors 35
4 OptimizationRBasics:RARMachineRLearningRView 47
5 OptimizationR ChallengesR andR AdvancedR Solutions 57
6 LagrangianR RelaxationR andR Duality 63
7 SingularR ValueR Decomposition 71
8 MatrixR Factorization 81
9 TheR LinearR AlgebraR ofR Similarity 89
10 TheR LinearR AlgebraR ofR Graphs 95
11 OptimizationR inR ComputationalR Graphs 101
viii
,ChapterR 1
LinearRAlgebraRandROptimization:RAnRIntroduction
1. ForR anyR twoR vectorsR xR andR y,R whichR areR eachR ofR lengthR a,R showR thatR (i
)R xR−RyR isRorthogonalRtoRxR+Ry,R andR(ii)R theRdotRproductRofRxR−R3yR andRxR
+R3yR isR negative.
(i)RTheRfirstRisRsimply·R −RRx·R xR yR yRusingRtheRdistributiveRpropertyRofRmatrix
Rmultiplication. RTheRdotRproduct RofRaRvector RwithRitselfRis Rits Rsquared Rle
ngth.RSinceRbothRvectorsRareRofRtheRsameRlength,RitRfollowsRthatRtheRresu
ltRisR0.R(ii)RInRtheRsecondRcase,RoneRcanRuseRaRsimilarRargumentRtoRshowRt
hatRtheRresultRisRa2R−R9a2,RwhichRisRnegative.
2. ConsiderR aR situationR inR whichR youR haveR threeR matricesR A,R B,R andR C,R ofR si
zesR 10R×R2,R2R×R10,RandR 10R×R10,R respectively.
(a) SupposeRyouRhadRtoRcomputeRtheRmatrixRproductRABC.RFromRanRefficie
ncyRper-
Rspective,RwouldRitRcomputationallyRmakeRmoreRsenseRtoRcomputeR(AB)CRo
rRwouldRitRmakeRmoreRsenseRtoRcomputeRA(BC)?
(b) IfRyouRhadRtoRcomputeRtheRmatrixRproductRCAB,RwouldRitRmakeRmoreRse
nseRtoRcomputeR (CA)BR orR C(AB)?
TheRmainRpointRisRtoRkeepRtheRsizeRofRtheRintermediateRmatrixRasRsm
allRasRpossibleR inRorderRtoRreduceRbothRcomputationalRandRspaceRrequ
irements.RInRtheRcaseRofRABC,RitRmakesRsenseRtoRcomputeRBCRfirst.RInR
theRcaseRofRCABRitRmakesRsenseRtoRcomputeRCARfirst.RThisRtypeRofRass
ociativityRpropertyRisRusedRfrequentlyRinRmachineRlearningRinRorderRt
oRreduceRcomputationalRrequirements.
3. ShowR thatR ifR aR matrixR AR satisfies—
R AR =
ATR,R thenR allR theR diagonalR elementsR of
R theRmatrixRareR0.
NoteRthatRAR+RATR=R0.RHowever,RthisRmatrixRalsoRcontainsRtwiceRtheR
diagonalRelementsRofRARonRitsRdiagonal.RTherefore,RtheRdiagonalRelem
entsRofRARmustRbeR0.
4. ShowRthatRifRweRhaveRaRmatrixRsatisfying
— RAR=
1
, ATR,RthenRforRanyRcolumnRvectorRx
,RweRhaveR x RAxR=R0.
T
NoteR thatR theR transposeR ofR theR scalarR xTRAxR remainsR unchanged.R Therefore,R
weR have
xTRAxR=R(xTRAx)TR =RxTRATRxR=R−xTRAx.R Therefore,R weR haveR 2xTRAxR=R
0.
2
Linear Algebra and Optimization for Machine Learning
1st Edition by Charu Aggarwal. Chapters 1–11
vii
,Contents
1 LinearR AlgebraR andR Optimization:R AnR Introduction 1
2 LinearR Transformations R andR LinearR Systems 17
3 Diagonalizable R MatricesR andR Eigenvectors 35
4 OptimizationRBasics:RARMachineRLearningRView 47
5 OptimizationR ChallengesR andR AdvancedR Solutions 57
6 LagrangianR RelaxationR andR Duality 63
7 SingularR ValueR Decomposition 71
8 MatrixR Factorization 81
9 TheR LinearR AlgebraR ofR Similarity 89
10 TheR LinearR AlgebraR ofR Graphs 95
11 OptimizationR inR ComputationalR Graphs 101
viii
,ChapterR 1
LinearRAlgebraRandROptimization:RAnRIntroduction
1. ForR anyR twoR vectorsR xR andR y,R whichR areR eachR ofR lengthR a,R showR thatR (i
)R xR−RyR isRorthogonalRtoRxR+Ry,R andR(ii)R theRdotRproductRofRxR−R3yR andRxR
+R3yR isR negative.
(i)RTheRfirstRisRsimply·R −RRx·R xR yR yRusingRtheRdistributiveRpropertyRofRmatrix
Rmultiplication. RTheRdotRproduct RofRaRvector RwithRitselfRis Rits Rsquared Rle
ngth.RSinceRbothRvectorsRareRofRtheRsameRlength,RitRfollowsRthatRtheRresu
ltRisR0.R(ii)RInRtheRsecondRcase,RoneRcanRuseRaRsimilarRargumentRtoRshowRt
hatRtheRresultRisRa2R−R9a2,RwhichRisRnegative.
2. ConsiderR aR situationR inR whichR youR haveR threeR matricesR A,R B,R andR C,R ofR si
zesR 10R×R2,R2R×R10,RandR 10R×R10,R respectively.
(a) SupposeRyouRhadRtoRcomputeRtheRmatrixRproductRABC.RFromRanRefficie
ncyRper-
Rspective,RwouldRitRcomputationallyRmakeRmoreRsenseRtoRcomputeR(AB)CRo
rRwouldRitRmakeRmoreRsenseRtoRcomputeRA(BC)?
(b) IfRyouRhadRtoRcomputeRtheRmatrixRproductRCAB,RwouldRitRmakeRmoreRse
nseRtoRcomputeR (CA)BR orR C(AB)?
TheRmainRpointRisRtoRkeepRtheRsizeRofRtheRintermediateRmatrixRasRsm
allRasRpossibleR inRorderRtoRreduceRbothRcomputationalRandRspaceRrequ
irements.RInRtheRcaseRofRABC,RitRmakesRsenseRtoRcomputeRBCRfirst.RInR
theRcaseRofRCABRitRmakesRsenseRtoRcomputeRCARfirst.RThisRtypeRofRass
ociativityRpropertyRisRusedRfrequentlyRinRmachineRlearningRinRorderRt
oRreduceRcomputationalRrequirements.
3. ShowR thatR ifR aR matrixR AR satisfies—
R AR =
ATR,R thenR allR theR diagonalR elementsR of
R theRmatrixRareR0.
NoteRthatRAR+RATR=R0.RHowever,RthisRmatrixRalsoRcontainsRtwiceRtheR
diagonalRelementsRofRARonRitsRdiagonal.RTherefore,RtheRdiagonalRelem
entsRofRARmustRbeR0.
4. ShowRthatRifRweRhaveRaRmatrixRsatisfying
— RAR=
1
, ATR,RthenRforRanyRcolumnRvectorRx
,RweRhaveR x RAxR=R0.
T
NoteR thatR theR transposeR ofR theR scalarR xTRAxR remainsR unchanged.R Therefore,R
weR have
xTRAxR=R(xTRAx)TR =RxTRATRxR=R−xTRAx.R Therefore,R weR haveR 2xTRAxR=R
0.
2
