CS7643 QUIZ 4 EXAM WITH CORRECT
ANSWERS
Embedding n- ncorrect nanswers n-A nlearned nmap nfrom nentities nto nvectors nthat
nencodes nsimilarity
Graph nEmbedding n- ncorrect nanswers n-Optimize nthe nobjective nthat nconnected
nnodes nhave nmore nsimilar nembeddings nthan nunconnected nnodes.
Task: nconvert nnodes nto nvectors
- neffectively nunsupervised nlearning nwhere nnearest nneighbors nare nsimilar
- nthese nlearned nvectors nare nuseful nfor ndownstream ntasks
Multi-layer nPerceptron n(MLP) npain npoints nfor nNLP n- ncorrect nanswers n-- nCannot
neasily nsupport nvariable-sized nsequences nas ninputs nor noutputs
- nNo ninherent ntemporal nstructure
, - nNo npractical nway nof nholding nstate
- nThe nsize nof nthe nnetwork ngrows nwith nthe nmaximum nallowed nsize nof nthe ninput
nor noutput nsequences
Truncated nBackpropagation nthrough ntime n- ncorrect nanswers n-- nOnly
nbackpropagate na nRNN nthrough nT ntime nsteps
Recurrent nNeural nNetworks n(RNN) n- ncorrect nanswers n-h(t) n= nactivation(U*input n+
nV*h(t-1) n+ nbias)
y(t) n= nactivation(W*h(t) n+ nbias)
- nactivation nis ntypically nthe nlogistic nfunction nor ntanh
- noutputs ncan nalso nsimply nbe nh(t)
- nfamily nof nNN narchitectures nfor nmodeling nsequences
Training nVanilla nRNN's ndifficulties n- ncorrect nanswers n-- nVanishing ngradients
- nSince ndx(t)/dx(t-1) n= nw^t
- nif nw n> n1: nexploding ngradients
- nif nw n< n1: nvanishing ngradients
Long nShort-Term nMemory nNetwork nGates nand nStates n- ncorrect nanswers n-- nf(t) n=
nforget ngate
- ni(t) n= ninput ngate
- nu(t) n= ncandidate nupdate ngate
- no(t) n= noutput ngate
- nc(t) n= ncell nstate
- nc(t) n= nf(t) n* nc(t n- n1) n+ ni(t) n* nu(t)
- nh(t) n= nhidden nstate
- nh(t) n= no(t) n* ntanh(c(t))
Perplexity(s) n- ncorrect nanswers n-= nproduct( n1 n/ nP(w(i) n| nw(i-1), n...) n) n^ n(1 n/ nN)
= nb n^ n(-1/N nsum( nlog(b) n(P(w(i) n| nw(i-1), n...) n) n)
- nnote nexponent nof nb nis nper nword nCE nloss
- nperplexity nof na ndiscrete nuniform ndistribution nover nk nevents nis nk
Language nModel nGoal n- ncorrect nanswers n-- nestimate nthe nprobability nof
nsequences nof nwords
- np(s) n= np(w1, nw2, n..., nwn)
Masked nLanguage nModeling n- ncorrect nanswers n-- npre-training ntask n- nan nauxiliary
ntask ndifferent nfrom nthe nfinal ntask nwe're nreally ninterested nin, nbut nwhich ncan nhelp
nus nachieve nbetter nperformance nfinding ngood ninitial nparameters nfor nthe nmodel
ANSWERS
Embedding n- ncorrect nanswers n-A nlearned nmap nfrom nentities nto nvectors nthat
nencodes nsimilarity
Graph nEmbedding n- ncorrect nanswers n-Optimize nthe nobjective nthat nconnected
nnodes nhave nmore nsimilar nembeddings nthan nunconnected nnodes.
Task: nconvert nnodes nto nvectors
- neffectively nunsupervised nlearning nwhere nnearest nneighbors nare nsimilar
- nthese nlearned nvectors nare nuseful nfor ndownstream ntasks
Multi-layer nPerceptron n(MLP) npain npoints nfor nNLP n- ncorrect nanswers n-- nCannot
neasily nsupport nvariable-sized nsequences nas ninputs nor noutputs
- nNo ninherent ntemporal nstructure
, - nNo npractical nway nof nholding nstate
- nThe nsize nof nthe nnetwork ngrows nwith nthe nmaximum nallowed nsize nof nthe ninput
nor noutput nsequences
Truncated nBackpropagation nthrough ntime n- ncorrect nanswers n-- nOnly
nbackpropagate na nRNN nthrough nT ntime nsteps
Recurrent nNeural nNetworks n(RNN) n- ncorrect nanswers n-h(t) n= nactivation(U*input n+
nV*h(t-1) n+ nbias)
y(t) n= nactivation(W*h(t) n+ nbias)
- nactivation nis ntypically nthe nlogistic nfunction nor ntanh
- noutputs ncan nalso nsimply nbe nh(t)
- nfamily nof nNN narchitectures nfor nmodeling nsequences
Training nVanilla nRNN's ndifficulties n- ncorrect nanswers n-- nVanishing ngradients
- nSince ndx(t)/dx(t-1) n= nw^t
- nif nw n> n1: nexploding ngradients
- nif nw n< n1: nvanishing ngradients
Long nShort-Term nMemory nNetwork nGates nand nStates n- ncorrect nanswers n-- nf(t) n=
nforget ngate
- ni(t) n= ninput ngate
- nu(t) n= ncandidate nupdate ngate
- no(t) n= noutput ngate
- nc(t) n= ncell nstate
- nc(t) n= nf(t) n* nc(t n- n1) n+ ni(t) n* nu(t)
- nh(t) n= nhidden nstate
- nh(t) n= no(t) n* ntanh(c(t))
Perplexity(s) n- ncorrect nanswers n-= nproduct( n1 n/ nP(w(i) n| nw(i-1), n...) n) n^ n(1 n/ nN)
= nb n^ n(-1/N nsum( nlog(b) n(P(w(i) n| nw(i-1), n...) n) n)
- nnote nexponent nof nb nis nper nword nCE nloss
- nperplexity nof na ndiscrete nuniform ndistribution nover nk nevents nis nk
Language nModel nGoal n- ncorrect nanswers n-- nestimate nthe nprobability nof
nsequences nof nwords
- np(s) n= np(w1, nw2, n..., nwn)
Masked nLanguage nModeling n- ncorrect nanswers n-- npre-training ntask n- nan nauxiliary
ntask ndifferent nfrom nthe nfinal ntask nwe're nreally ninterested nin, nbut nwhich ncan nhelp
nus nachieve nbetter nperformance nfinding ngood ninitial nparameters nfor nthe nmodel
