GEORGIA TECH CS 7643 DEEP LEARNING QUIZ 3
PRACTICE | TRANSFORMER & ATTENTION QUESTIONS,
ANSWERS & DETAILED RATIONALES | FALL 2026
149 Questions with Answers and Detailed Rationales
100 PERCENT GUARANTEED PASS
INSTANT DOWNLOAD ANSWERS INCLUDED
IMPORTANCE OF THIS DOCUMENT
This comprehensive examination preparation guide has been meticulously developed to help you succeed in the
GEORGIA TECH CS 7643 DEEP LEARNING QUIZ 3 PRACTICE | TRANSFORMER & ATTENTION
QUESTIONS, ANSWERS & DETAILED RATIONALES | FALL 2026. It contains 149 carefully selected questions
that reflect the most current exam content and testing strategies. Each question is accompanied by a correct
answer and a detailed rationale that explains the underlying pathophysiology, pharmacology, or clinical reasoning.
Self-Assessment – Test your knowledge and Exam Preparation – Familiarize yourself with the
identify areas requiring further question format and content
study areas
Concept Reinforcement – Deepen your Confidence Building – Develop test-taking
understanding through strategies and reduce
evidence-based exam anxiety
rationales
Time Management – Practice answering
questions under simulated
exam conditions
Review Summary 149 Questions
Foundations - Application - Georgia TECH CS 7643 DEEP Learning 3 Transformer & Attention & Detailed
Rationales FALL 2026 DEEP Learning Transformers Self-attention AND Sequence Modeling Graduate M S
/ph D Level R1 Research University
All answers with rationales
,Table of Contents
Content Area Questions Key Topics
Attention Mechanisms AND 1-25 Transformer, Attention, Dimension, Model, Softmax
Self-attention
Transformer Architecture 26-50 Attention, Transformer, Model, Positional, Applied
AND Components
Positional Encoding AND 51-75 Transformer, Attention, Dimension, Layer, Model
Sequence Representation
Multi-head Attention AND 76-100 Attention, Transformer, Layer, Model, Applied
Scaled Dot-product Attention
Encoder-decoder 101-125 Attention, Model, Transformer, Dimension, Positional
Architectures AND
Sequence-to-sequence
Models
Training Transformers 126-149 Attention, Transformer, Model, Token, Softmax
Optimization Regularization
AND Learning RATE
Schedules
TOTAL 149 All questions include answers and detailed rationales
,Section A - Attention Mechanisms AND Self-attention
Q1.
In scaled dot-product attention, why is the dot product divided by sqrt(d_k) before the
softmax?
A. To reduce the number of parameters in B. To prevent the softmax from saturating
the attention layer due to large dot-product magnitudes as d_k
grows
C. To enforce that attention weights sum to D. To make the attention operation
exactly one across heads permutation-equivariant
Correct: B - To prevent the softmax from saturating due to large dot-product magnitudes
as d_k grows
Rationale:For large d_k, the variance of the dot product grows with d_k, pushing softmax
inputs into regions with tiny gradients and causing saturation. Dividing by sqrt(d_k) stabilizes
the variance and keeps gradients well-behaved. The other options misattribute the scaling's
purpose to parameter counts, normalization across heads, or permutation equivariance.
Why the other answers are wrong:
A. Scaling does not change the number of parameters; it only rescales logits.
C. Softmax already normalizes weights to sum to one per query; scaling is unrelated.
D. Permutation equivariance comes from the attention structure itself, not the scaling factor.
Reference: Vaswani et al. (2017). Attention Is All You Need, NeurIPS.
Q2.
A decoder-only transformer is trained with causal masking. Which statement best
describes the effect of the mask on the attention score matrix?
A. It zeros out attention weights for future B. It sets logits for future positions to a large
positions after softmax negative value before softmax
C. It removes future tokens from the D. It averages attention weights
key/value matrices entirely symmetrically across all positions
Correct: B - It sets logits for future positions to a large negative value before softmax
Rationale:Causal masking adds -infinity (or a large negative constant) to the attention logits
for future positions before softmax, ensuring those positions receive zero weight. Zeroing
after softmax (A) would break normalization, and the mask does not physically remove tokens
(C) or symmetrize weights (D).
Why the other answers are wrong:
A. Zeroing after softmax would leave weights unnormalized and is not how masking is
implemented.
Page 3
, Section A - Attention Mechanisms AND Self-attention
C. The key/value matrices still contain all tokens; masking only affects logits.
D. Causal masking is asymmetric by design, not symmetric averaging.
Reference: Radford et al. (2019). Language Models are Unsupervised Multitask Learners, OpenAI.
Q3.
Which of the following is the primary reason sinusoidal positional encodings are added
rather than concatenated to token embeddings in the original transformer?
A. Concatenation would double the model's B. Addition preserves dimensionality and
parameter count allows the model to learn relative position
via linear projections
C. Addition guarantees the embeddings D. Concatenation prevents the use of layer
remain orthogonal normalization
Correct: B - Addition preserves dimensionality and allows the model to learn relative
position via linear projections
Rationale:Adding positional encodings keeps the embedding dimension unchanged and lets
subsequent linear projections exploit the sinusoidal structure to infer relative positions.
Concatenation would increase dimensionality and complicate the architecture; it does not
inherently double parameters (A) or block layer norm (D).
Why the other answers are wrong:
A. Concatenation increases input dimension but does not necessarily double parameters.
C. Addition does not guarantee orthogonality of embeddings.
D. Layer normalization can still be applied after concatenation.
Reference: Vaswani et al. (2017). Attention Is All You Need, NeurIPS.
Q4.
In multi-head attention with h heads and model dimension d_model, what is the typical
dimension of each head's key and query projections?
A. d_model B. d_model / h
C. h * d_model D. sqrt(d_model)
Correct: B - d_model / h
Rationale:Each head projects Q, K, V into a lower-dimensional subspace of size d_model / h
so that the total computation remains comparable to single-head attention. Using d_model (A)
or h * d_model (C) would blow up compute, and sqrt(d_model) (D) is not the standard
convention.
Why the other answers are wrong:
A. Using full d_model per head multiplies total compute by h.
C. h * d_model would drastically increase parameter count and compute.
Page 4
PRACTICE | TRANSFORMER & ATTENTION QUESTIONS,
ANSWERS & DETAILED RATIONALES | FALL 2026
149 Questions with Answers and Detailed Rationales
100 PERCENT GUARANTEED PASS
INSTANT DOWNLOAD ANSWERS INCLUDED
IMPORTANCE OF THIS DOCUMENT
This comprehensive examination preparation guide has been meticulously developed to help you succeed in the
GEORGIA TECH CS 7643 DEEP LEARNING QUIZ 3 PRACTICE | TRANSFORMER & ATTENTION
QUESTIONS, ANSWERS & DETAILED RATIONALES | FALL 2026. It contains 149 carefully selected questions
that reflect the most current exam content and testing strategies. Each question is accompanied by a correct
answer and a detailed rationale that explains the underlying pathophysiology, pharmacology, or clinical reasoning.
Self-Assessment – Test your knowledge and Exam Preparation – Familiarize yourself with the
identify areas requiring further question format and content
study areas
Concept Reinforcement – Deepen your Confidence Building – Develop test-taking
understanding through strategies and reduce
evidence-based exam anxiety
rationales
Time Management – Practice answering
questions under simulated
exam conditions
Review Summary 149 Questions
Foundations - Application - Georgia TECH CS 7643 DEEP Learning 3 Transformer & Attention & Detailed
Rationales FALL 2026 DEEP Learning Transformers Self-attention AND Sequence Modeling Graduate M S
/ph D Level R1 Research University
All answers with rationales
,Table of Contents
Content Area Questions Key Topics
Attention Mechanisms AND 1-25 Transformer, Attention, Dimension, Model, Softmax
Self-attention
Transformer Architecture 26-50 Attention, Transformer, Model, Positional, Applied
AND Components
Positional Encoding AND 51-75 Transformer, Attention, Dimension, Layer, Model
Sequence Representation
Multi-head Attention AND 76-100 Attention, Transformer, Layer, Model, Applied
Scaled Dot-product Attention
Encoder-decoder 101-125 Attention, Model, Transformer, Dimension, Positional
Architectures AND
Sequence-to-sequence
Models
Training Transformers 126-149 Attention, Transformer, Model, Token, Softmax
Optimization Regularization
AND Learning RATE
Schedules
TOTAL 149 All questions include answers and detailed rationales
,Section A - Attention Mechanisms AND Self-attention
Q1.
In scaled dot-product attention, why is the dot product divided by sqrt(d_k) before the
softmax?
A. To reduce the number of parameters in B. To prevent the softmax from saturating
the attention layer due to large dot-product magnitudes as d_k
grows
C. To enforce that attention weights sum to D. To make the attention operation
exactly one across heads permutation-equivariant
Correct: B - To prevent the softmax from saturating due to large dot-product magnitudes
as d_k grows
Rationale:For large d_k, the variance of the dot product grows with d_k, pushing softmax
inputs into regions with tiny gradients and causing saturation. Dividing by sqrt(d_k) stabilizes
the variance and keeps gradients well-behaved. The other options misattribute the scaling's
purpose to parameter counts, normalization across heads, or permutation equivariance.
Why the other answers are wrong:
A. Scaling does not change the number of parameters; it only rescales logits.
C. Softmax already normalizes weights to sum to one per query; scaling is unrelated.
D. Permutation equivariance comes from the attention structure itself, not the scaling factor.
Reference: Vaswani et al. (2017). Attention Is All You Need, NeurIPS.
Q2.
A decoder-only transformer is trained with causal masking. Which statement best
describes the effect of the mask on the attention score matrix?
A. It zeros out attention weights for future B. It sets logits for future positions to a large
positions after softmax negative value before softmax
C. It removes future tokens from the D. It averages attention weights
key/value matrices entirely symmetrically across all positions
Correct: B - It sets logits for future positions to a large negative value before softmax
Rationale:Causal masking adds -infinity (or a large negative constant) to the attention logits
for future positions before softmax, ensuring those positions receive zero weight. Zeroing
after softmax (A) would break normalization, and the mask does not physically remove tokens
(C) or symmetrize weights (D).
Why the other answers are wrong:
A. Zeroing after softmax would leave weights unnormalized and is not how masking is
implemented.
Page 3
, Section A - Attention Mechanisms AND Self-attention
C. The key/value matrices still contain all tokens; masking only affects logits.
D. Causal masking is asymmetric by design, not symmetric averaging.
Reference: Radford et al. (2019). Language Models are Unsupervised Multitask Learners, OpenAI.
Q3.
Which of the following is the primary reason sinusoidal positional encodings are added
rather than concatenated to token embeddings in the original transformer?
A. Concatenation would double the model's B. Addition preserves dimensionality and
parameter count allows the model to learn relative position
via linear projections
C. Addition guarantees the embeddings D. Concatenation prevents the use of layer
remain orthogonal normalization
Correct: B - Addition preserves dimensionality and allows the model to learn relative
position via linear projections
Rationale:Adding positional encodings keeps the embedding dimension unchanged and lets
subsequent linear projections exploit the sinusoidal structure to infer relative positions.
Concatenation would increase dimensionality and complicate the architecture; it does not
inherently double parameters (A) or block layer norm (D).
Why the other answers are wrong:
A. Concatenation increases input dimension but does not necessarily double parameters.
C. Addition does not guarantee orthogonality of embeddings.
D. Layer normalization can still be applied after concatenation.
Reference: Vaswani et al. (2017). Attention Is All You Need, NeurIPS.
Q4.
In multi-head attention with h heads and model dimension d_model, what is the typical
dimension of each head's key and query projections?
A. d_model B. d_model / h
C. h * d_model D. sqrt(d_model)
Correct: B - d_model / h
Rationale:Each head projects Q, K, V into a lower-dimensional subspace of size d_model / h
so that the total computation remains comparable to single-head attention. Using d_model (A)
or h * d_model (C) would blow up compute, and sqrt(d_model) (D) is not the standard
convention.
Why the other answers are wrong:
A. Using full d_model per head multiplies total compute by h.
C. h * d_model would drastically increase parameter count and compute.
Page 4
