SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearx Algebrax andx Optimization:x Anx Introduction 1
2 Linearx Transformationsx andx Linearx Systems 17
3 Diagonalizablex Matricesx andx Eigenvectors 35
4 OptimizationxBasics:xAxMachinexLearningxView 47
5 Optimizationx Challengesx andx Advancedx Solutions 57
6 Lagrangianx Relaxationx andx Duality 63
7 Singularx Valuex Decomposition 71
8 Matrixx Factorization 81
9 Thex Linearx Algebrax ofx Similarity 89
10 Thex Linearx Algebrax ofx Graphs 95
11 Optimizationx inx Computationalx Graphs 101
viii
,Chapterx 1
LinearxAlgebraxandxOptimization:xAnxIntroduction
1. Forx anyx twox vectorsx xx andx y,x whichx arex eachx ofx lengthx a,x showx thatx (i)x
xx−xyx isxorthogonalxtoxxx+xy,x andx(ii)x thexdotxproductxofxxx−x3yx andxxx+x3
yx isx negative.
(i)xThexfirstxisxsimply
·x −xx
x ·x x x yx yxusingxthexdistributivexpropertyxofxmatrixx
multiplication.xThexdotxproductxofxaxvectorxwithxitselfxisxitsxsquaredxlen
gth.xSincexbothxvectorsxarexofxthexsamexlength,xitxfollowsxthatxthexresult
xisx0.x(ii)xInxthexsecondxcase,xonexcanxusexaxsimilarxargumentxtoxshowxtha
txthexresultxisxa2x−x9a2,xwhichxisxnegative.
2. Considerx ax situationx inx whichx youx havex threex matricesx A,x B,x andx C,x ofx size
sx 10x×x2,x2x×x10,xandx10x×x10,xrespectively.
(a) SupposexyouxhadxtoxcomputexthexmatrixxproductxABC.xFromxanxefficien
cyxper-
xspective,xwouldxitxcomputationallyxmakexmorexsensextoxcomputex(AB)Cxor
xwouldxit xmake xmore xsense xtoxcompute xA(BC)?
(b) IfxyouxhadxtoxcomputexthexmatrixxproductxCAB,xwouldxitxmakexmorexse
nsextoxcomputex (CA)Bx orx C(AB)?
Thexmainxpointxisxtoxkeepxthexsizexofxthexintermediatexmatrixxasxsma
llxasxpossiblex inxorderxtoxreducexbothxcomputationalxandxspacexrequir
ements.xInxthexcasexofxABC,xitxmakesxsensextoxcomputexBCxfirst.xInxth
excasexofxCABxitxmakesxsensextoxcomputexCAxfirst.xThisxtypexofxassoci
ativityxpropertyxisxusedxfrequentlyxinxmachinexlearningxinxorderxtoxre
ducexcomputationalxrequirements.
3. Showx thatx ifx ax matrixx Ax satisfiesx—Ax =
ATx,x thenx allx thex diagonalx elementsx of
x the xmatrix xare x0.
NotexthatxAx+xATx=x0.xHowever,xthisxmatrixxalsoxcontainsxtwicexthexd
iagonalxelementsxofxAxonxitsxdiagonal.xTherefore,xthexdiagonalxeleme
ntsxofxAxmustxbex0.
4. Showxthatxifxwexhavexaxmatrixxsatisfying
— xAx=
1
, ATx,xthenxforxanyxcolumnxvectorxx,
wexhavex x xAxx=x0.
x
T
Notex thatx thex transposex ofx thex scalarx xTxAxx remainsx unchanged.x Therefore,x
wex have
xTxAxx=x(xTxAx)Tx =xxTxATxxx=x−xTxAx.x Therefore,x wex havex 2xTxAxx=x0.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 Linearx Algebrax andx Optimization:x Anx Introduction 1
2 Linearx Transformationsx andx Linearx Systems 17
3 Diagonalizablex Matricesx andx Eigenvectors 35
4 OptimizationxBasics:xAxMachinexLearningxView 47
5 Optimizationx Challengesx andx Advancedx Solutions 57
6 Lagrangianx Relaxationx andx Duality 63
7 Singularx Valuex Decomposition 71
8 Matrixx Factorization 81
9 Thex Linearx Algebrax ofx Similarity 89
10 Thex Linearx Algebrax ofx Graphs 95
11 Optimizationx inx Computationalx Graphs 101
viii
,Chapterx 1
LinearxAlgebraxandxOptimization:xAnxIntroduction
1. Forx anyx twox vectorsx xx andx y,x whichx arex eachx ofx lengthx a,x showx thatx (i)x
xx−xyx isxorthogonalxtoxxx+xy,x andx(ii)x thexdotxproductxofxxx−x3yx andxxx+x3
yx isx negative.
(i)xThexfirstxisxsimply
·x −xx
x ·x x x yx yxusingxthexdistributivexpropertyxofxmatrixx
multiplication.xThexdotxproductxofxaxvectorxwithxitselfxisxitsxsquaredxlen
gth.xSincexbothxvectorsxarexofxthexsamexlength,xitxfollowsxthatxthexresult
xisx0.x(ii)xInxthexsecondxcase,xonexcanxusexaxsimilarxargumentxtoxshowxtha
txthexresultxisxa2x−x9a2,xwhichxisxnegative.
2. Considerx ax situationx inx whichx youx havex threex matricesx A,x B,x andx C,x ofx size
sx 10x×x2,x2x×x10,xandx10x×x10,xrespectively.
(a) SupposexyouxhadxtoxcomputexthexmatrixxproductxABC.xFromxanxefficien
cyxper-
xspective,xwouldxitxcomputationallyxmakexmorexsensextoxcomputex(AB)Cxor
xwouldxit xmake xmore xsense xtoxcompute xA(BC)?
(b) IfxyouxhadxtoxcomputexthexmatrixxproductxCAB,xwouldxitxmakexmorexse
nsextoxcomputex (CA)Bx orx C(AB)?
Thexmainxpointxisxtoxkeepxthexsizexofxthexintermediatexmatrixxasxsma
llxasxpossiblex inxorderxtoxreducexbothxcomputationalxandxspacexrequir
ements.xInxthexcasexofxABC,xitxmakesxsensextoxcomputexBCxfirst.xInxth
excasexofxCABxitxmakesxsensextoxcomputexCAxfirst.xThisxtypexofxassoci
ativityxpropertyxisxusedxfrequentlyxinxmachinexlearningxinxorderxtoxre
ducexcomputationalxrequirements.
3. Showx thatx ifx ax matrixx Ax satisfiesx—Ax =
ATx,x thenx allx thex diagonalx elementsx of
x the xmatrix xare x0.
NotexthatxAx+xATx=x0.xHowever,xthisxmatrixxalsoxcontainsxtwicexthexd
iagonalxelementsxofxAxonxitsxdiagonal.xTherefore,xthexdiagonalxeleme
ntsxofxAxmustxbex0.
4. Showxthatxifxwexhavexaxmatrixxsatisfying
— xAx=
1
, ATx,xthenxforxanyxcolumnxvectorxx,
wexhavex x xAxx=x0.
x
T
Notex thatx thex transposex ofx thex scalarx xTxAxx remainsx unchanged.x Therefore,x
wex have
xTxAxx=x(xTxAx)Tx =xxTxATxxx=x−xTxAx.x Therefore,x wex havex 2xTxAxx=x0.
2
