CS-7643 Quiz 4 Exam – Deep Learning Optimization & Regularization Study Guide
Embedding - (ANSWER)A learned map from entities to vectors that encodes similarity
Graph Embedding - (ANSWER)Optimize the objective that connected nodes have more similar
embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - (ANSWER)- Cannot easily support variable-sized
sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output sequences
Truncated Backpropagation through time - (ANSWER)- Only backpropagate a RNN through T time steps
Recurrent Neural Networks (RNN) - (ANSWER)h(t) = activation(U*input + V*h(t-1) + bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - (ANSWER)- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
, CS-7643 Quiz 4 Exam – Deep Learning Optimization & Regularization Study Guide
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - (ANSWER)- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
- c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
Perplexity(s) - (ANSWER)= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - (ANSWER)- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - (ANSWER)- pre-training task - an auxiliary task different from the final task
we're really interested in, but which can help us achieve better performance finding good initial
parameters for the model
- By pre-training on masked language modeling before training on our final task, it is usually possible to
obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - (ANSWER)- Have fully parameterized teacher model
Embedding - (ANSWER)A learned map from entities to vectors that encodes similarity
Graph Embedding - (ANSWER)Optimize the objective that connected nodes have more similar
embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - (ANSWER)- Cannot easily support variable-sized
sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output sequences
Truncated Backpropagation through time - (ANSWER)- Only backpropagate a RNN through T time steps
Recurrent Neural Networks (RNN) - (ANSWER)h(t) = activation(U*input + V*h(t-1) + bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - (ANSWER)- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
, CS-7643 Quiz 4 Exam – Deep Learning Optimization & Regularization Study Guide
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - (ANSWER)- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
- c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
Perplexity(s) - (ANSWER)= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - (ANSWER)- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - (ANSWER)- pre-training task - an auxiliary task different from the final task
we're really interested in, but which can help us achieve better performance finding good initial
parameters for the model
- By pre-training on masked language modeling before training on our final task, it is usually possible to
obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - (ANSWER)- Have fully parameterized teacher model
