CS7643 QUIZ 5 - P2 WITH VERIFIED
SOLUTIONS
Discriminative ovs. oGenerative oModels o- ocorrect oanswers o-1. oDiscriminative omodels
omodel othe oconditional odistribution, oprobability oof olabel ogiven oinput oP(y|x)
- oNN, oSVM, oetc.
- oNo oway oto ohandle ounreasonable ooutputs o(i.e. oif otrained oon ocats oand odog
oimages, owill oalways ooutput oa olabel oof ocat oor odog)
2. oGenerative oModels omodel odistribution oover oinput ospace oP(x)
- oCan oparameterize oour omodel oas oP(x, otheta) oand ouse oMaximum oLikelihood
oEstimation oto ooptimize othe oparameters
- oCalled ogenerative obecause othey ocan ogenerate osamples
- oCan o"reject" ounreasonable oinputs oas obeing otoo ounlikely
- oFeature oLearning owithout olabels
, Explicit ovs oImplicit oDensity o- ocorrect oanswers o-Explicit: oExplicitly odefine oand
osolve ofor oPmodel(x|θ), ogiven oan oexplicit olikelihood othat ocan obe omaximized
Implicit: oLearn oa oparameterized ogeneration omodel othat ocan osample ofrom othe ojoint
odistribution oPmodel(x|θ) owithout oexplicitly odefining oPmodel(x|θ)
Tractable oDensity oEstimation o- ocorrect oanswers o-- oExplicit oDensity
- oSimplify ojoint odistribution ointo ofactorized omodel oof osimpler ocomponents oand
ooptimize owith orespect oto osimpler ocomponents
- oPixelRNN/PixelCNN
Approximate oDensity oEstimation o- ocorrect oanswers o-- oLearn odistributions othat
oapproximate otrue ojoint odistribution
- oVariational oAutoencoder
- oWe odon't omodel odensity oitself, obut ocan osample ofrom othe odistribution
- oGAN
Implicit oDensity oEstimation o- ocorrect oanswers o-- oWe odon't omodel odensity oitself,
obut ocan osample ofrom othe odistribution
- oGAN
Factorized oModels ofor oImages oDownsides o- ocorrect oanswers o-1. oSlow osequential
ogeneration oprocess
2. oOnly oconsiders ofew ocontext opixels
PixelRNN o- ocorrect oanswers o-Explicit o& oTractable
Uses ochain orule oto odecompose othe olikelihood oof oan oimage ox ointo oa oproduct oof
o1-d odistributions
Similar oto olanguage omodels
Requires oordering oof ovariables!
Training: opixel odepends oimplicitly oon othe opixels oabove oand oto othe oleft. o
Generating: oGenerates opixels oone oat oa otime, overy oslow, oonly oconsiders othe
opixels oimmediately oadjacent owhen ogenerating
Pixel oCNN o- ocorrect oanswers o-Idea: oRepresent oconditional odistribution oas oa
oconvolution olayer
- oConsiders olarger ocontext o(receptive ofield)
- oPractically ocan obe oimplemented oby oapplying oa omask, ozeroing oout o"future"
opixels
SOLUTIONS
Discriminative ovs. oGenerative oModels o- ocorrect oanswers o-1. oDiscriminative omodels
omodel othe oconditional odistribution, oprobability oof olabel ogiven oinput oP(y|x)
- oNN, oSVM, oetc.
- oNo oway oto ohandle ounreasonable ooutputs o(i.e. oif otrained oon ocats oand odog
oimages, owill oalways ooutput oa olabel oof ocat oor odog)
2. oGenerative oModels omodel odistribution oover oinput ospace oP(x)
- oCan oparameterize oour omodel oas oP(x, otheta) oand ouse oMaximum oLikelihood
oEstimation oto ooptimize othe oparameters
- oCalled ogenerative obecause othey ocan ogenerate osamples
- oCan o"reject" ounreasonable oinputs oas obeing otoo ounlikely
- oFeature oLearning owithout olabels
, Explicit ovs oImplicit oDensity o- ocorrect oanswers o-Explicit: oExplicitly odefine oand
osolve ofor oPmodel(x|θ), ogiven oan oexplicit olikelihood othat ocan obe omaximized
Implicit: oLearn oa oparameterized ogeneration omodel othat ocan osample ofrom othe ojoint
odistribution oPmodel(x|θ) owithout oexplicitly odefining oPmodel(x|θ)
Tractable oDensity oEstimation o- ocorrect oanswers o-- oExplicit oDensity
- oSimplify ojoint odistribution ointo ofactorized omodel oof osimpler ocomponents oand
ooptimize owith orespect oto osimpler ocomponents
- oPixelRNN/PixelCNN
Approximate oDensity oEstimation o- ocorrect oanswers o-- oLearn odistributions othat
oapproximate otrue ojoint odistribution
- oVariational oAutoencoder
- oWe odon't omodel odensity oitself, obut ocan osample ofrom othe odistribution
- oGAN
Implicit oDensity oEstimation o- ocorrect oanswers o-- oWe odon't omodel odensity oitself,
obut ocan osample ofrom othe odistribution
- oGAN
Factorized oModels ofor oImages oDownsides o- ocorrect oanswers o-1. oSlow osequential
ogeneration oprocess
2. oOnly oconsiders ofew ocontext opixels
PixelRNN o- ocorrect oanswers o-Explicit o& oTractable
Uses ochain orule oto odecompose othe olikelihood oof oan oimage ox ointo oa oproduct oof
o1-d odistributions
Similar oto olanguage omodels
Requires oordering oof ovariables!
Training: opixel odepends oimplicitly oon othe opixels oabove oand oto othe oleft. o
Generating: oGenerates opixels oone oat oa otime, overy oslow, oonly oconsiders othe
opixels oimmediately oadjacent owhen ogenerating
Pixel oCNN o- ocorrect oanswers o-Idea: oRepresent oconditional odistribution oas oa
oconvolution olayer
- oConsiders olarger ocontext o(receptive ofield)
- oPractically ocan obe oimplemented oby oapplying oa omask, ozeroing oout o"future"
opixels
