SOLUTION6MANUAL
LINEAR6ALGEBRA6AND6OPTIMIZATION6FOR6MAC
HINE6LEARNING
1ST6EDITION6BY6CHARU6AGGARWAL.66CHAPTER
S616–611
vii
,Contents
viii
,1 Linear6 Algebra6 and6 Optimization:6 An6 Introduction 1
2 Linear6 Transformations6and6 Linear6 Systems 17
3 Diagonalizable6Matrices6and6Eigenvectors 35
4 Optimization6Basics:6A6Machine6Learning6View 47
5 Optimization6Challenges6and6Advanced6Solutions 57
6 Lagrangian6 Relaxation6 and6 Duality 63
7 Singular6 Value6 Decomposition 71
8 Matrix6 Factorization 81
9 The6 Linear6 Algebra6 of6 Similarity 89
10 The6 Linear6 Algebra6 of6 Graphs 95
11 Optimization6 in6 Computational6 Graphs 101
ix
, Chapter6 1
Linear6Algebra6and6Optimization:6An6Introduction
1. For6 any6 two6 vectors6 x6 and6 y,6 which6 are6 each6 of6 length6 a,6 show
6 that6 (i)6 x6−6y6 is6orthogonal6to6x6+6y,6and6(ii)6the6dot6product6
of6x6−63y6 and6x6+63y6 is6negative.
(i)6The6first6is6simply6x6
·6 − x6 y6 y6using6the6distributive6property6of
6 ·
6matrix6multiplication.6The6dot6product6of6a6vector6with6itself6is6i
ts6squared6length.6Since6both6vectors6are6of6the6same6length,6it6f
ollows6that6the6result6is60.6(ii)6In6the6second6case,6one6can6use6a
6similar6argument6to6show6that6the6result6is6a26−69a2,6which6is6n
egative.
2. Consider6a6situation6in6which6you6have6three6matrices6A,6B,6and6C
,6of6sizes6106×62,626×610,6and6106×610,6respectively.
(a) Suppose6you6had6to6compute6the6matrix6product6ABC.6From6an6
efficiency6per-
6spective,6would6it6computationally6make6more6sense6to6compute6(A
B)C6or6would6it6make6more6sense6to6compute6A(BC)?
(b) If6you6had6to6compute6the6matrix6product6CAB,6would6it6make6
more6sense6to6compute6 (CA)B6 or6 C(AB)?
The6main6point6is6to6keep6the6size6of6the6intermediate6matrix6
as6small6as6possible6 in6order6to6reduce6both6computational6and
6space6requirements.6In6the6case6of6ABC,6it6makes6sense6to6co
mpute6BC6first.6In6the6case6of6CAB6it6makes6sense6to6compute
6CA6first.6This6type6of6associativity6property6is6used6frequently
6in6machine6learning6in6order6to6reduce6computational6requirem
ents.
3. —
Show6 that6 if6 a6 matrix6 A6 satisfies6 A6 =
A 6,6then6all6the6diagonal6element
T
s6of6the6matrix6are60.
Note6that6A6+6AT6=60.6However,6this6matrix6also6contains6twi
ce6the6diagonal6elements6of6A6on6its6diagonal.6Therefore,6the6d
iagonal6elements6of6A6must6be60.
1
LINEAR6ALGEBRA6AND6OPTIMIZATION6FOR6MAC
HINE6LEARNING
1ST6EDITION6BY6CHARU6AGGARWAL.66CHAPTER
S616–611
vii
,Contents
viii
,1 Linear6 Algebra6 and6 Optimization:6 An6 Introduction 1
2 Linear6 Transformations6and6 Linear6 Systems 17
3 Diagonalizable6Matrices6and6Eigenvectors 35
4 Optimization6Basics:6A6Machine6Learning6View 47
5 Optimization6Challenges6and6Advanced6Solutions 57
6 Lagrangian6 Relaxation6 and6 Duality 63
7 Singular6 Value6 Decomposition 71
8 Matrix6 Factorization 81
9 The6 Linear6 Algebra6 of6 Similarity 89
10 The6 Linear6 Algebra6 of6 Graphs 95
11 Optimization6 in6 Computational6 Graphs 101
ix
, Chapter6 1
Linear6Algebra6and6Optimization:6An6Introduction
1. For6 any6 two6 vectors6 x6 and6 y,6 which6 are6 each6 of6 length6 a,6 show
6 that6 (i)6 x6−6y6 is6orthogonal6to6x6+6y,6and6(ii)6the6dot6product6
of6x6−63y6 and6x6+63y6 is6negative.
(i)6The6first6is6simply6x6
·6 − x6 y6 y6using6the6distributive6property6of
6 ·
6matrix6multiplication.6The6dot6product6of6a6vector6with6itself6is6i
ts6squared6length.6Since6both6vectors6are6of6the6same6length,6it6f
ollows6that6the6result6is60.6(ii)6In6the6second6case,6one6can6use6a
6similar6argument6to6show6that6the6result6is6a26−69a2,6which6is6n
egative.
2. Consider6a6situation6in6which6you6have6three6matrices6A,6B,6and6C
,6of6sizes6106×62,626×610,6and6106×610,6respectively.
(a) Suppose6you6had6to6compute6the6matrix6product6ABC.6From6an6
efficiency6per-
6spective,6would6it6computationally6make6more6sense6to6compute6(A
B)C6or6would6it6make6more6sense6to6compute6A(BC)?
(b) If6you6had6to6compute6the6matrix6product6CAB,6would6it6make6
more6sense6to6compute6 (CA)B6 or6 C(AB)?
The6main6point6is6to6keep6the6size6of6the6intermediate6matrix6
as6small6as6possible6 in6order6to6reduce6both6computational6and
6space6requirements.6In6the6case6of6ABC,6it6makes6sense6to6co
mpute6BC6first.6In6the6case6of6CAB6it6makes6sense6to6compute
6CA6first.6This6type6of6associativity6property6is6used6frequently
6in6machine6learning6in6order6to6reduce6computational6requirem
ents.
3. —
Show6 that6 if6 a6 matrix6 A6 satisfies6 A6 =
A 6,6then6all6the6diagonal6element
T
s6of6the6matrix6are60.
Note6that6A6+6AT6=60.6However,6this6matrix6also6contains6twi
ce6the6diagonal6elements6of6A6on6its6diagonal.6Therefore,6the6d
iagonal6elements6of6A6must6be60.
1
