CS 7643 — Quiz 2 Study Guide
Optimization (Lesson 3) + Convolution & Pooling (Lesson 5)
Conceptual topics — graphs, explanations, and memory tricks
Quiz: June 15, 8:00 AM ET — NO GRACE PERIOD (closed notes/book/internet)
Key idea
The spine of the whole quiz: “Keep the signal alive — not too small, not too dead.”
Almost every topic is a fix for the gradient/signal dying or exploding. For any scenario, ask
two questions: (1) is the gradient too small to learn from? (2) is this an activation fault or
an initialization fault?
Contents
1 Activation Functions 2
1.1 Saturation ⇒ vanishing gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The ReLU family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Initialization (Xavier vs. He) 4
3 Loss Functions 4
3.1 Why CE beats MSE for classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4 Optimizers 5
5 Batch Size Effects (DL Book §8.1.3) 7
6 Batch Normalization (DL Book §8.7.1) 7
7 Regularization & Dropout 8
7.1 Dropout — the star . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8 Supervised Pre-training (DL Book §8.7.4) 9
9 Convolution Concepts (DL Book Ch. 9) 9
9.1 The three motivations (S–P–E) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.2 Pooling, equivariance vs. invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.3 Computation (already drilled) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
10 Symptom → Cause Reflex (the skill the quiz tests) 11
1
, 1 Activation Functions
The whole comparison game scores each function on four properties: saturation (does the gra-
dient vanish for large |x|?), zero-centered output (all-positive outputs cause zig-zag updates),
gradient magnitude (small ⇒ slow learning), and compute cost (exp is expensive, max is
cheap).
Function Range Saturates? Zero-centered? Key issue
1
Sigmoid σ(x) = 1+e−x
(0, 1) yes No vanishing (max slope 0.25), zig-zag, exp cost
Tanh (−1, 1) yes Yes still vanishes, but fixes zig-zag
ReLU max(0, x) [0, ∞) no (x>0) no dead neurons (x<0 ⇒ out 0, grad 0)
Leaky ReLU max(αx, x) (−∞, ∞) no ∼no small slope α revives dead neurons
2
Optimization (Lesson 3) + Convolution & Pooling (Lesson 5)
Conceptual topics — graphs, explanations, and memory tricks
Quiz: June 15, 8:00 AM ET — NO GRACE PERIOD (closed notes/book/internet)
Key idea
The spine of the whole quiz: “Keep the signal alive — not too small, not too dead.”
Almost every topic is a fix for the gradient/signal dying or exploding. For any scenario, ask
two questions: (1) is the gradient too small to learn from? (2) is this an activation fault or
an initialization fault?
Contents
1 Activation Functions 2
1.1 Saturation ⇒ vanishing gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The ReLU family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Initialization (Xavier vs. He) 4
3 Loss Functions 4
3.1 Why CE beats MSE for classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4 Optimizers 5
5 Batch Size Effects (DL Book §8.1.3) 7
6 Batch Normalization (DL Book §8.7.1) 7
7 Regularization & Dropout 8
7.1 Dropout — the star . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8 Supervised Pre-training (DL Book §8.7.4) 9
9 Convolution Concepts (DL Book Ch. 9) 9
9.1 The three motivations (S–P–E) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.2 Pooling, equivariance vs. invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.3 Computation (already drilled) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
10 Symptom → Cause Reflex (the skill the quiz tests) 11
1
, 1 Activation Functions
The whole comparison game scores each function on four properties: saturation (does the gra-
dient vanish for large |x|?), zero-centered output (all-positive outputs cause zig-zag updates),
gradient magnitude (small ⇒ slow learning), and compute cost (exp is expensive, max is
cheap).
Function Range Saturates? Zero-centered? Key issue
1
Sigmoid σ(x) = 1+e−x
(0, 1) yes No vanishing (max slope 0.25), zig-zag, exp cost
Tanh (−1, 1) yes Yes still vanishes, but fixes zig-zag
ReLU max(0, x) [0, ∞) no (x>0) no dead neurons (x<0 ⇒ out 0, grad 0)
Leaky ReLU max(αx, x) (−∞, ∞) no ∼no small slope α revives dead neurons
2
