Georgia Tech CS 7643 Deep Learning Quiz 2 Complete Practice & Review
Bundle | 56 Questions, Answers & Detailed Rationales | Fall 2026
Question 1
Which issues commonly make optimization difficult in deep neural networks? Select all that
apply.
A. Local minima are always the dominant problem
B. Poorly conditioned loss surfaces
C. Noisy gradient estimates
D. Saddle points
Answer: B, C, D
Rationale: Ill-conditioned curvature can cause slow or unstable progress because gradients
behave very differently across directions. Mini-batch training introduces stochastic gradient
noise, while saddle points can create regions with very small gradients and slow optimization.
The source does not key local minima as one of the selected answers.
Question 2
What is a main advantage of Leaky ReLU over ordinary ReLU?
A. It never saturates for positive inputs
B. It guarantees positive outputs
C. It is always cheaper computationally
D. It retains a nonzero gradient for negative inputs
Answer: D
Rationale: Standard ReLU outputs zero for negative inputs and therefore has zero derivative
there. A neuron can become effectively inactive if it remains in this region. Leaky ReLU assigns
a small negative-side slope, allowing some gradient to continue flowing.
Question 3
Which activation functions are most strongly associated with the classic vanishing-gradient
problem?
,A. ReLU
B. Sigmoid and tanh
C. Linear functions
D. Max pooling
Answer: B
Rationale: Sigmoid and tanh can enter saturated regions where their derivatives become very
small. Across many layers, multiplying many small derivatives can make the gradient shrink
dramatically.
Question 4
Which description best matches batch normalization?
A. It deliberately adds gradient noise
B. It normalizes intermediate activations using mini-batch statistics to help stabilize training
C. It automatically increases the learning rate
D. It randomly deletes neurons
Answer: B
Rationale: Batch normalization standardizes activations using batch statistics and then applies
learned scale and shift parameters. It can improve optimization stability and often permits more
aggressive learning rates.
Question 5
Why can stochastic or mini-batch gradient descent sometimes optimize deep networks more
effectively than full-batch gradient descent?
A. It guarantees a global optimum
B. It uses Hessian information
C. Stochasticity can help optimization move away from unfavorable flat or saddle regions
D. It always needs fewer epochs
Answer: C
Rationale: Mini-batch gradients contain noise because they estimate the full gradient from only
part of the training set. This noise can sometimes help the optimizer avoid getting stuck for long
periods in difficult regions of the loss landscape.
,Question 6
What is the primary purpose of momentum in gradient descent?
A. To eliminate every source of gradient variance
B. To accumulate information from past updates so movement becomes more consistent in
useful directions
C. To deliberately slow optimization
D. To normalize the learning rate
Answer: B
Rationale: Momentum maintains a running velocity based on previous gradients. This can
accelerate movement along directions where gradients consistently agree while reducing
oscillation across steep directions.
Question 7
Which optimizer combines momentum-like first-moment estimates with adaptive per-parameter
learning rates?
A. RMSProp
B. AdaGrad
C. Adam
D. Plain SGD
Answer: C
Rationale: Adam tracks both the first moment of gradients and the second moment of squared
gradients. These estimates are used to scale parameter updates adaptively.
Question 8
Which technique is commonly used to control exploding gradients?
A. Weight initialization alone
B. Gradient clipping
C. Dropout
D. L2 regularization
, Answer: B
Rationale: Gradient clipping limits gradient magnitude, typically by clipping individual values
or rescaling the gradient when its norm exceeds a threshold. This prevents extremely large
updates from destabilizing training.
Question 9
What is the main goal of Xavier/Glorot initialization?
A. Make all weights positive
B. Keep activation and gradient scales reasonably stable across layers
C. Remove the need for normalization
D. Initialize all weights to zero
Answer: B
Rationale: Xavier initialization chooses weight variance based on layer size so that signals do
not systematically explode or vanish as they propagate through the network. It is especially
associated with approximately symmetric activations such as tanh.
Question 10
Why are residual connections useful in very deep networks?
A. They prevent all underfitting
B. They provide shorter paths for information and gradients to flow through the network
C. They always reduce the parameter count
D. They eliminate backpropagation
Answer: B
Rationale: A residual block often learns
y=F(x)+x.y = F(x)+x.
The identity path allows signals and gradients to bypass multiple nonlinear transformations,
helping very deep models optimize more effectively.
Question 11
Bundle | 56 Questions, Answers & Detailed Rationales | Fall 2026
Question 1
Which issues commonly make optimization difficult in deep neural networks? Select all that
apply.
A. Local minima are always the dominant problem
B. Poorly conditioned loss surfaces
C. Noisy gradient estimates
D. Saddle points
Answer: B, C, D
Rationale: Ill-conditioned curvature can cause slow or unstable progress because gradients
behave very differently across directions. Mini-batch training introduces stochastic gradient
noise, while saddle points can create regions with very small gradients and slow optimization.
The source does not key local minima as one of the selected answers.
Question 2
What is a main advantage of Leaky ReLU over ordinary ReLU?
A. It never saturates for positive inputs
B. It guarantees positive outputs
C. It is always cheaper computationally
D. It retains a nonzero gradient for negative inputs
Answer: D
Rationale: Standard ReLU outputs zero for negative inputs and therefore has zero derivative
there. A neuron can become effectively inactive if it remains in this region. Leaky ReLU assigns
a small negative-side slope, allowing some gradient to continue flowing.
Question 3
Which activation functions are most strongly associated with the classic vanishing-gradient
problem?
,A. ReLU
B. Sigmoid and tanh
C. Linear functions
D. Max pooling
Answer: B
Rationale: Sigmoid and tanh can enter saturated regions where their derivatives become very
small. Across many layers, multiplying many small derivatives can make the gradient shrink
dramatically.
Question 4
Which description best matches batch normalization?
A. It deliberately adds gradient noise
B. It normalizes intermediate activations using mini-batch statistics to help stabilize training
C. It automatically increases the learning rate
D. It randomly deletes neurons
Answer: B
Rationale: Batch normalization standardizes activations using batch statistics and then applies
learned scale and shift parameters. It can improve optimization stability and often permits more
aggressive learning rates.
Question 5
Why can stochastic or mini-batch gradient descent sometimes optimize deep networks more
effectively than full-batch gradient descent?
A. It guarantees a global optimum
B. It uses Hessian information
C. Stochasticity can help optimization move away from unfavorable flat or saddle regions
D. It always needs fewer epochs
Answer: C
Rationale: Mini-batch gradients contain noise because they estimate the full gradient from only
part of the training set. This noise can sometimes help the optimizer avoid getting stuck for long
periods in difficult regions of the loss landscape.
,Question 6
What is the primary purpose of momentum in gradient descent?
A. To eliminate every source of gradient variance
B. To accumulate information from past updates so movement becomes more consistent in
useful directions
C. To deliberately slow optimization
D. To normalize the learning rate
Answer: B
Rationale: Momentum maintains a running velocity based on previous gradients. This can
accelerate movement along directions where gradients consistently agree while reducing
oscillation across steep directions.
Question 7
Which optimizer combines momentum-like first-moment estimates with adaptive per-parameter
learning rates?
A. RMSProp
B. AdaGrad
C. Adam
D. Plain SGD
Answer: C
Rationale: Adam tracks both the first moment of gradients and the second moment of squared
gradients. These estimates are used to scale parameter updates adaptively.
Question 8
Which technique is commonly used to control exploding gradients?
A. Weight initialization alone
B. Gradient clipping
C. Dropout
D. L2 regularization
, Answer: B
Rationale: Gradient clipping limits gradient magnitude, typically by clipping individual values
or rescaling the gradient when its norm exceeds a threshold. This prevents extremely large
updates from destabilizing training.
Question 9
What is the main goal of Xavier/Glorot initialization?
A. Make all weights positive
B. Keep activation and gradient scales reasonably stable across layers
C. Remove the need for normalization
D. Initialize all weights to zero
Answer: B
Rationale: Xavier initialization chooses weight variance based on layer size so that signals do
not systematically explode or vanish as they propagate through the network. It is especially
associated with approximately symmetric activations such as tanh.
Question 10
Why are residual connections useful in very deep networks?
A. They prevent all underfitting
B. They provide shorter paths for information and gradients to flow through the network
C. They always reduce the parameter count
D. They eliminate backpropagation
Answer: B
Rationale: A residual block often learns
y=F(x)+x.y = F(x)+x.
The identity path allows signals and gradients to bypass multiple nonlinear transformations,
helping very deep models optimize more effectively.
Question 11
