CS7643 Quiz 4 – 2025/2026 Updated Study
Guide & Practice Questions | 100% Pass
Guarantee
Embedding - correct answerA learned map from entities to vectors that encodes
similarity
Graph Embedding - correct answerOptimize the objective that connected nodes have
more similar embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - correct answer- Cannot easily
support variable-sized sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output
sequences
Truncated Backpropagation through time - correct answer- Only backpropagate a RNN
through T time steps
Recurrent Neural Networks (RNN) - correct answerh(t) = activation(U*input + V*h(t-1) +
bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - correct answer- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - correct answer- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
, - c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
Perplexity(s) - correct answer= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - correct answer- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - correct answer- pre-training task - an auxiliary task
different from the final task we're really interested in, but which can help us achieve
better performance finding good initial parameters for the model
- By pre-training on masked language modeling before training on our final task, it is
usually possible to obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - correct answer- Have fully
parameterized teacher model
- Have a much smaller student model
- Student model attempts to minimize prediction error and distance to teacher model
simultaneously
L(dist) = CE b/w student and teacher predictions
L(student) = CE b/w predicted output and actual
L = alpha * L(dist) + beta * L(student)
Advantages:
- may work well b/c of soft predictions of teacher model
- if we don't have enough labeled text we can still train student model to align
predictions
Collobert and Weston Vector Idea - correct answera word and its context is a positive
training sample; a random word in that sample context gives a negative training sample
Word2vec Overview - correct answerWord2vec - a framework for learning word vector
Idea:
- we have a large corpus of text
- every word in. fixed vocabulary represented by a vector
- Go through each position t in the text, which has a center word c and context words o
Guide & Practice Questions | 100% Pass
Guarantee
Embedding - correct answerA learned map from entities to vectors that encodes
similarity
Graph Embedding - correct answerOptimize the objective that connected nodes have
more similar embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - correct answer- Cannot easily
support variable-sized sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output
sequences
Truncated Backpropagation through time - correct answer- Only backpropagate a RNN
through T time steps
Recurrent Neural Networks (RNN) - correct answerh(t) = activation(U*input + V*h(t-1) +
bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - correct answer- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - correct answer- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
, - c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
Perplexity(s) - correct answer= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - correct answer- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - correct answer- pre-training task - an auxiliary task
different from the final task we're really interested in, but which can help us achieve
better performance finding good initial parameters for the model
- By pre-training on masked language modeling before training on our final task, it is
usually possible to obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - correct answer- Have fully
parameterized teacher model
- Have a much smaller student model
- Student model attempts to minimize prediction error and distance to teacher model
simultaneously
L(dist) = CE b/w student and teacher predictions
L(student) = CE b/w predicted output and actual
L = alpha * L(dist) + beta * L(student)
Advantages:
- may work well b/c of soft predictions of teacher model
- if we don't have enough labeled text we can still train student model to align
predictions
Collobert and Weston Vector Idea - correct answera word and its context is a positive
training sample; a random word in that sample context gives a negative training sample
Word2vec Overview - correct answerWord2vec - a framework for learning word vector
Idea:
- we have a large corpus of text
- every word in. fixed vocabulary represented by a vector
- Go through each position t in the text, which has a center word c and context words o
