Quiz 2 — Practice Set (26 Conceptual Questions)
CS7643 Deep Learning — Optimization & Regularization
How to use this set. Cover the answer key (Section 2) and work the questions cold, writing
A/B/C/D for each. Then check and read the one-line reason for any you miss.
Scope caveat. These are conceptual practice questions compiled from a third-party set. A few
topics — ResNets, learning-rate warm-up / cosine annealing, gradient clipping, the Hessian — go
beyond the official Ed #98 checklist, and this set has no computation questions (the real quiz
is ∼15% output-size / parameter-count math). Treat it as supplementary drill, not a scope match.
“Select all that apply” on Q1.
1 Questions
Q1 — Common optimization issues (select all that apply)
A. Existence of local minima
B. Ill-conditioned loss surface
C. Noisy gradient estimates
D. Saddle points
Q2 — Advantage of Leaky ReLU vs ReLU
A. There’s no saturation on the positive end
B. Its output is always positive
C. It is cheap to compute
D. There’s no “dead” neuron when computing gradients
Q3 — The vanishing gradient problem occurs primarily with
A. ReLU activations
B. Tanh and Sigmoid activations
C. Linear transformations
D. Max pooling layers
, Q4 — Which best describes batch normalization?
A. Adds noise to gradients during backpropagation
B. Normalizes inputs within a mini-batch to stabilize training
C. Increases the learning rate dynamically
D. Removes neurons to prevent overfitting
Q5 — Why does SGD often converge better than full-batch GD?
A. It guarantees global minima
B. It uses second-order derivatives
C. The noise helps escape saddle points and local minima
D. It requires fewer epochs
Q6 — The role of momentum in gradient descent
A. To reduce variance in gradients
B. To accumulate past gradients to speed up convergence and escape local minima
C. To slow down convergence for stability
D. To normalize the learning rate
Q7 — Which optimizer combines momentum with adaptive per-parameter learning
rates?
A. RMSProp
B. AdaGrad
C. Adam
D. SGD
Q8 — Exploding gradients are most commonly addressed by
A. Weight initialization
B. Gradient clipping
C. Dropout
D. Adding regularization
Q9 — Xavier/Glorot initialization is designed to
A. Ensure weights are all positive
B. Keep the variance of activations consistent across layers
C. Reduce training time by skipping normalization
D. Initialize all weights at zero
2
CS7643 Deep Learning — Optimization & Regularization
How to use this set. Cover the answer key (Section 2) and work the questions cold, writing
A/B/C/D for each. Then check and read the one-line reason for any you miss.
Scope caveat. These are conceptual practice questions compiled from a third-party set. A few
topics — ResNets, learning-rate warm-up / cosine annealing, gradient clipping, the Hessian — go
beyond the official Ed #98 checklist, and this set has no computation questions (the real quiz
is ∼15% output-size / parameter-count math). Treat it as supplementary drill, not a scope match.
“Select all that apply” on Q1.
1 Questions
Q1 — Common optimization issues (select all that apply)
A. Existence of local minima
B. Ill-conditioned loss surface
C. Noisy gradient estimates
D. Saddle points
Q2 — Advantage of Leaky ReLU vs ReLU
A. There’s no saturation on the positive end
B. Its output is always positive
C. It is cheap to compute
D. There’s no “dead” neuron when computing gradients
Q3 — The vanishing gradient problem occurs primarily with
A. ReLU activations
B. Tanh and Sigmoid activations
C. Linear transformations
D. Max pooling layers
, Q4 — Which best describes batch normalization?
A. Adds noise to gradients during backpropagation
B. Normalizes inputs within a mini-batch to stabilize training
C. Increases the learning rate dynamically
D. Removes neurons to prevent overfitting
Q5 — Why does SGD often converge better than full-batch GD?
A. It guarantees global minima
B. It uses second-order derivatives
C. The noise helps escape saddle points and local minima
D. It requires fewer epochs
Q6 — The role of momentum in gradient descent
A. To reduce variance in gradients
B. To accumulate past gradients to speed up convergence and escape local minima
C. To slow down convergence for stability
D. To normalize the learning rate
Q7 — Which optimizer combines momentum with adaptive per-parameter learning
rates?
A. RMSProp
B. AdaGrad
C. Adam
D. SGD
Q8 — Exploding gradients are most commonly addressed by
A. Weight initialization
B. Gradient clipping
C. Dropout
D. Adding regularization
Q9 — Xavier/Glorot initialization is designed to
A. Ensure weights are all positive
B. Keep the variance of activations consistent across layers
C. Reduce training time by skipping normalization
D. Initialize all weights at zero
2
