SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
,Contents
1 Linear6 Algebra6 and6 Optimization:6 An6 Introduction 1
2 Linear6 Transformations6 and6 Linear6 Systems 17
3 Diagonalizable 6 Matrices6 and6 Eigenvectors 35
4 Optimization6Basics:6A6Machine6Learning6View 47
5 Optimization6 Challenges6 and6 Advanced6 Solutions 57
6 Lagrangian6 Relaxation6 and6 Duality 63
7 Singular6 Value6 Decomposition 71
8 Matrix6 Factorization 81
9 The6 Linear6 Algebra6 of6 Similarity 89
10 The6 Linear6 Algebra6 of6 Graphs 95
11 Optimization6 in6 Computational6 Graphs 101
,Chapter6 1
Linear6Algebra6and6Optimization:6An6Introduction
1. For6 any6 two6 vectors6 x6 and6 y,6 which6 are6 each6 of6 length6 a,6 show6 that6 (i)6 x6
−6y6 is6orthogonal6to6x6+6y,6 and6(ii)6 the6dot6product6of6x6−63y6 and6x6+63y6 is6
negative.
(i)6The6first6is6simply
·6 6−x66 ·x6 y6 y6using6the6distributive6property6of6matrix6m
ultiplication.6The6dot6product6of6a6vector6with6itself6is6its6squared6length.6
Since6both6vectors6are6of6the6same6length,6it6follows6that6the6result6is60.6(ii
)6In6the6second6case,6one6can6use6a6similar6argument6to6show6that6the6resu
lt6is6a26−69a2,6which6is6negative.
2. Consider6 a6 situation6 in6 which6 you6 have6 three6 matrices6 A,6 B,6 and6 C,6 of6 sizes6
106×62,626×610,6and6106×610,6respectively.
(a) Suppose6you6had6to6compute6the6matrix6product6ABC.6From6an6efficienc
y6per-
6spective,6would6it6computationally6make6more6sense6to6compute6(AB)C6or6
would6it6make6more6sense6to6compute6A(BC)?
(b) If6you6had6to6compute6the6matrix6product6CAB,6would6it6make6more6sens
e6to6compute6 (CA)B6 or6 C(AB)?
The6main6point6is6to6keep6the6size6of6the6intermediate6matrix6as6small6
as6possible6 in6order6to6reduce6both6computational6and6space6requirem
ents.6In6the6case6of6ABC,6it6makes6sense6to6compute6BC6first.6In6the6cas
e6of6CAB6it6makes6sense6to6compute6CA6first.6This6type6of6associativity
6property6is6used 6frequently6in6machine6learning6in6order6to6reduce6co
mputational6requirements.
3. Show6 that6 if6 a6 matrix6 A6 satisfies6 A6 =
AT6,6 then6 all6 the6 diagonal6 elements6 of6 t
he6matrix6are60.
Note6that6A6+6AT6=60.6However,6this6matrix6also6contains6twice6the6dia
gonal6elements6of6A6on6its6diagonal.6Therefore,6the6diagonal6elements6
of6A6must6be60.
4. Show6that6if6we6have6a6matrix6satisfying6A6=
1
, AT6,6then6for6any6column6vector6x,6
we6have6 x 6Ax6=60.
T
Note6 that6 the6 transpose6 of6 the6 scalar6 xT6Ax6 remains6 unchanged.6 Therefore,6 we
6 have
xT6Ax6=6(xT6Ax)T6 =6xT6AT6x6=6−xT6Ax.6 Therefore,6 we6 have6 2xT6Ax6=60.
2
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
,Contents
1 Linear6 Algebra6 and6 Optimization:6 An6 Introduction 1
2 Linear6 Transformations6 and6 Linear6 Systems 17
3 Diagonalizable 6 Matrices6 and6 Eigenvectors 35
4 Optimization6Basics:6A6Machine6Learning6View 47
5 Optimization6 Challenges6 and6 Advanced6 Solutions 57
6 Lagrangian6 Relaxation6 and6 Duality 63
7 Singular6 Value6 Decomposition 71
8 Matrix6 Factorization 81
9 The6 Linear6 Algebra6 of6 Similarity 89
10 The6 Linear6 Algebra6 of6 Graphs 95
11 Optimization6 in6 Computational6 Graphs 101
,Chapter6 1
Linear6Algebra6and6Optimization:6An6Introduction
1. For6 any6 two6 vectors6 x6 and6 y,6 which6 are6 each6 of6 length6 a,6 show6 that6 (i)6 x6
−6y6 is6orthogonal6to6x6+6y,6 and6(ii)6 the6dot6product6of6x6−63y6 and6x6+63y6 is6
negative.
(i)6The6first6is6simply
·6 6−x66 ·x6 y6 y6using6the6distributive6property6of6matrix6m
ultiplication.6The6dot6product6of6a6vector6with6itself6is6its6squared6length.6
Since6both6vectors6are6of6the6same6length,6it6follows6that6the6result6is60.6(ii
)6In6the6second6case,6one6can6use6a6similar6argument6to6show6that6the6resu
lt6is6a26−69a2,6which6is6negative.
2. Consider6 a6 situation6 in6 which6 you6 have6 three6 matrices6 A,6 B,6 and6 C,6 of6 sizes6
106×62,626×610,6and6106×610,6respectively.
(a) Suppose6you6had6to6compute6the6matrix6product6ABC.6From6an6efficienc
y6per-
6spective,6would6it6computationally6make6more6sense6to6compute6(AB)C6or6
would6it6make6more6sense6to6compute6A(BC)?
(b) If6you6had6to6compute6the6matrix6product6CAB,6would6it6make6more6sens
e6to6compute6 (CA)B6 or6 C(AB)?
The6main6point6is6to6keep6the6size6of6the6intermediate6matrix6as6small6
as6possible6 in6order6to6reduce6both6computational6and6space6requirem
ents.6In6the6case6of6ABC,6it6makes6sense6to6compute6BC6first.6In6the6cas
e6of6CAB6it6makes6sense6to6compute6CA6first.6This6type6of6associativity
6property6is6used 6frequently6in6machine6learning6in6order6to6reduce6co
mputational6requirements.
3. Show6 that6 if6 a6 matrix6 A6 satisfies6 A6 =
AT6,6 then6 all6 the6 diagonal6 elements6 of6 t
he6matrix6are60.
Note6that6A6+6AT6=60.6However,6this6matrix6also6contains6twice6the6dia
gonal6elements6of6A6on6its6diagonal.6Therefore,6the6diagonal6elements6
of6A6must6be60.
4. Show6that6if6we6have6a6matrix6satisfying6A6=
1
, AT6,6then6for6any6column6vector6x,6
we6have6 x 6Ax6=60.
T
Note6 that6 the6 transpose6 of6 the6 scalar6 xT6Ax6 remains6 unchanged.6 Therefore,6 we
6 have
xT6Ax6=6(xT6Ax)T6 =6xT6AT6x6=6−xT6Ax.6 Therefore,6 we6 have6 2xT6Ax6=60.
2
