CS7643 - QUIZ 5 WITH FULL PACK SOLUTIONS
1. Properties vof vthe vsoftmax vfunction: v1.) vSoftmax vis vpermutation vequivariant
v(permutation vof vinput vleads vto vthe vsame vpermutation vof voutput)
2.) Softmax vis vnot vlinear
3.) Softmax vis vditterentiable
4.) Softmax vinterpolates vbetween va vdistribution vthat vselects vindex vuniformly vat vrandom vand
vdist vthat vselects vthe vargmax vindex vwith vprobability v1.
2. Attention: v weighting vor vprobability vdistribution vover vinputs vthat vdepends von vcomputational
vstate vand vinputs
3. what v does v softmax v allow?: v allows vrandom vsampling vof vnumbers vfrom voriginal vset vwi
va vdistribution
that vdepends von vthe vvalues vfrom vthe vnumbers vof vthe voriginal vset.
4. Softmax vAttention: v- vGiven va vset. vof vvectors v"U" vand va vquery vvector v"Q"
-We vcan vselect vthe vmost vsimilar vvector vto v"Q" vvia vp=softmax(U v* vQ)
1 v/
v4
1. Properties vof vthe vsoftmax vfunction: v1.) vSoftmax vis vpermutation vequivariant
v(permutation vof vinput vleads vto vthe vsame vpermutation vof voutput)
2.) Softmax vis vnot vlinear
3.) Softmax vis vditterentiable
4.) Softmax vinterpolates vbetween va vdistribution vthat vselects vindex vuniformly vat vrandom vand
vdist vthat vselects vthe vargmax vindex vwith vprobability v1.
2. Attention: v weighting vor vprobability vdistribution vover vinputs vthat vdepends von vcomputational
vstate vand vinputs
3. what v does v softmax v allow?: v allows vrandom vsampling vof vnumbers vfrom voriginal vset vwi
va vdistribution
that vdepends von vthe vvalues vfrom vthe vnumbers vof vthe voriginal vset.
4. Softmax vAttention: v- vGiven va vset. vof vvectors v"U" vand va vquery vvector v"Q"
-We vcan vselect vthe vmost vsimilar vvector vto v"Q" vvia vp=softmax(U v* vQ)
1 v/
v4
