CS7643 Quiz 7 Exam – Deep Learning Study Guide & Practice Solutions
Discriminative vs. Generative Models - (ANSWER)1. Discriminative models model P(y|x)
- NN, SVM, etc.
- No way to handle unreasonable outputs (i.e. if trained on cats and dog images, will always output a
label of cat or dog)
2. Generative Models model P(x)
- Can parameterize our model as P(x, theta) and use Maximum Likelihood Estimation to optimize the
parameters
- Called generative because they can generate samples
- Can "reject" unreasonable inputs as being too unlikely
- Feature Learning without labels
Explicit vs Implicit Density - (ANSWER)Explicit: Explicitly define and solve for Pmodel(x|θ), given an
explicit likelihood that can be maximized
Implicit: Learn a parameterized generation model that can sample from the joint distribution
Pmodel(x|θ) without explicitly defining Pmodel(x|θ)
Tractable Density Estimation - (ANSWER)- Explicit Density
- Simplify joint distribution into factorized model of simpler components and optimize with respect to
simpler components
- PixelRNN/PixelCNN
Approximate Density Estimation - (ANSWER)- Explicit Density
- Learn distributions that approximate true joint distribution
- Variational Autoencoder
Implicit Density Estimation - (ANSWER)- We don't model density itself, but can sample from the
distribution
- GAN
, CS7643 Quiz 7 Exam – Deep Learning Study Guide & Practice Solutions
Factorized Models for Images Downsides - (ANSWER)1. Slow sequential generation process
2. Only considers few context pixels
PixelRNN - (ANSWER)Explicit & Tractable
Uses chain rule to decompose the likelihood of an image x into a product of 1-d distributions
Similar to language models
Requires ordering of variables!
Training: pixel depends implicitly on the pixels above and to the left.
Generating: Generates pixels one at a time, very slow, only considers the pixels immediately adjacent
when generating
Pixel CNN - (ANSWER)Idea: Represent conditional distribution as a convolution layer
- Considers larger context (receptive field)
- Practically can be implemented by applying a mask, zeroing out "future" pixels
Disadvantages:
- Faster training, but still slow generation
- Limited to smaller images
PixelRNN and PixelCNN Recap - (ANSWER)Explicit density model, optimizes exact likelihood, good
samples. But inefficient sequential generation
Discriminative vs. Generative Models - (ANSWER)1. Discriminative models model P(y|x)
- NN, SVM, etc.
- No way to handle unreasonable outputs (i.e. if trained on cats and dog images, will always output a
label of cat or dog)
2. Generative Models model P(x)
- Can parameterize our model as P(x, theta) and use Maximum Likelihood Estimation to optimize the
parameters
- Called generative because they can generate samples
- Can "reject" unreasonable inputs as being too unlikely
- Feature Learning without labels
Explicit vs Implicit Density - (ANSWER)Explicit: Explicitly define and solve for Pmodel(x|θ), given an
explicit likelihood that can be maximized
Implicit: Learn a parameterized generation model that can sample from the joint distribution
Pmodel(x|θ) without explicitly defining Pmodel(x|θ)
Tractable Density Estimation - (ANSWER)- Explicit Density
- Simplify joint distribution into factorized model of simpler components and optimize with respect to
simpler components
- PixelRNN/PixelCNN
Approximate Density Estimation - (ANSWER)- Explicit Density
- Learn distributions that approximate true joint distribution
- Variational Autoencoder
Implicit Density Estimation - (ANSWER)- We don't model density itself, but can sample from the
distribution
- GAN
, CS7643 Quiz 7 Exam – Deep Learning Study Guide & Practice Solutions
Factorized Models for Images Downsides - (ANSWER)1. Slow sequential generation process
2. Only considers few context pixels
PixelRNN - (ANSWER)Explicit & Tractable
Uses chain rule to decompose the likelihood of an image x into a product of 1-d distributions
Similar to language models
Requires ordering of variables!
Training: pixel depends implicitly on the pixels above and to the left.
Generating: Generates pixels one at a time, very slow, only considers the pixels immediately adjacent
when generating
Pixel CNN - (ANSWER)Idea: Represent conditional distribution as a convolution layer
- Considers larger context (receptive field)
- Practically can be implemented by applying a mask, zeroing out "future" pixels
Disadvantages:
- Faster training, but still slow generation
- Limited to smaller images
PixelRNN and PixelCNN Recap - (ANSWER)Explicit density model, optimizes exact likelihood, good
samples. But inefficient sequential generation
