“CS7643 Quiz 4 Comprehensive Study Guide: Embeddings,
Graph and Word Representations, RNNs, LSTMs, Skip-Gram
Word2Vec, Masked Language Modeling, Knowledge Distillation,
t-SNE, Teacher Forcing, Conditional Language Models, and
Evaluation Metrics”
Embedding
QUESTION What is an embedding?
A: A learned map from entities to vectors that encodes similarity.
100%CORRET ANSWER: Embeddings allow similar entities to be close in vector
space.
Graph Embedding
QUESTION Purpose of graph embeddings?
A: Optimize the objective that connected nodes have more similar
embeddings than unconnected nodes.
100%CORRET ANSWER: Converts graph nodes into vectors useful for
downstream tasks.
QUESTION Why useful?
A: Task-agnostic entity representations; nearest neighbors are semantically
meaningful.
100%CORRET ANSWER: Works even with limited labeled data.
MLP Pain Points for NLP
QUESTION Why are MLPs limited for NLP?
A: Cannot handle variable-length sequences, no temporal structure, no
memory, size grows with max sequence length.
100%CORRET ANSWER: Sequences require context and state, which MLPs lack.
, QUESTION
A:
Truncated Backpropagation Through Time (TBPTT)
What is TBPTT?
Only backpropagate an RNN through T time steps.
100%CORRET ANSWER: Reduces computational cost and mitigates gradient
issues.
Recurrent Neural Networks (RNN)
QUESTION RNN update equations? A:
• h(t) = activation(U x(t) + V h(t-1) + bias)
• y(t) = activation(W h(t) + bias)
100%CORRET ANSWER: Recursively updates hidden state based on input
and previous hidden state.
QUESTION Training difficulties?
A: Vanishing and exploding gradients.
100%CORRET ANSWER: Multiplicative effects over time steps make long-term
dependencies hard.
Long Short-Term Memory (LSTM) Networks
QUESTION LSTM gates and states? A:
• f(t) = forget gate
• i(t) = input gate
• u(t) = candidate update gate
• o(t) = output gate
• c(t) = f(t)c(t-1) + i(t)u(t)
• h(t) = o(t) * tanh(c(t))
100%CORRET ANSWER: LSTM gates control memory flow to solve
vanishing gradient problem.
Graph and Word Representations, RNNs, LSTMs, Skip-Gram
Word2Vec, Masked Language Modeling, Knowledge Distillation,
t-SNE, Teacher Forcing, Conditional Language Models, and
Evaluation Metrics”
Embedding
QUESTION What is an embedding?
A: A learned map from entities to vectors that encodes similarity.
100%CORRET ANSWER: Embeddings allow similar entities to be close in vector
space.
Graph Embedding
QUESTION Purpose of graph embeddings?
A: Optimize the objective that connected nodes have more similar
embeddings than unconnected nodes.
100%CORRET ANSWER: Converts graph nodes into vectors useful for
downstream tasks.
QUESTION Why useful?
A: Task-agnostic entity representations; nearest neighbors are semantically
meaningful.
100%CORRET ANSWER: Works even with limited labeled data.
MLP Pain Points for NLP
QUESTION Why are MLPs limited for NLP?
A: Cannot handle variable-length sequences, no temporal structure, no
memory, size grows with max sequence length.
100%CORRET ANSWER: Sequences require context and state, which MLPs lack.
, QUESTION
A:
Truncated Backpropagation Through Time (TBPTT)
What is TBPTT?
Only backpropagate an RNN through T time steps.
100%CORRET ANSWER: Reduces computational cost and mitigates gradient
issues.
Recurrent Neural Networks (RNN)
QUESTION RNN update equations? A:
• h(t) = activation(U x(t) + V h(t-1) + bias)
• y(t) = activation(W h(t) + bias)
100%CORRET ANSWER: Recursively updates hidden state based on input
and previous hidden state.
QUESTION Training difficulties?
A: Vanishing and exploding gradients.
100%CORRET ANSWER: Multiplicative effects over time steps make long-term
dependencies hard.
Long Short-Term Memory (LSTM) Networks
QUESTION LSTM gates and states? A:
• f(t) = forget gate
• i(t) = input gate
• u(t) = candidate update gate
• o(t) = output gate
• c(t) = f(t)c(t-1) + i(t)u(t)
• h(t) = o(t) * tanh(c(t))
100%CORRET ANSWER: LSTM gates control memory flow to solve
vanishing gradient problem.
