DEEP LEARNING OPTIMIZATION AND
REGULARIZATION COMPLETE EXAM
REVIEW WITH FULL SOLUTIONS
●● How do you distinguish underfitting from overfitting in practice?
Answer: Look at the gap between train and validation loss. Underfitting:
both are high and close together (model hasn't learned). Overfitting:
train loss is low but validation loss is high (model memorised noise).
●● What is regularisation actually doing?
Answer: It injects a prior to constrain the hypothesis space. It trades a
small amount of training fit for better generalisation, preventing the
model from chasing data quirks. L2 pushes for small weights, L1 for
sparsity, augmentation encodes invariances.
●● What does the optimiser landscape look like and why does SGD
work?
Answer: In high dimensions, it's dominated by saddle points, not local
minima; there's usually an escape route. SGD works because its noisy
mini-batch estimate acts as an implicit regulariser, biasing the optimiser
away from sharp minima and toward flat, wide ones that generalise
better.
●● Why do neural networks learn at all?
, Answer: Hierarchical feature learning (edges -> parts -> objects)
combined with backpropagation, which systematically distributes
credit/blame so every parameter gets a personalised, local update signal.
●● What's the real effect of batch size?
Answer: Larger batches give a lower-variance gradient estimate (faster
compute), but often generalise worse because they remove the implicit
regularisation of small-batch gradient noise, converging to sharper
minima.
●● What problem does ReLU solve and what is the 'dying' problem?
Answer: It solves the vanishing gradient problem of sigmoid/tanh by
keeping a constant gradient of 1 for positive inputs. The 'dying' problem
occurs if a large update pushes a unit permanently below zero, killing its
gradient forever (fixed by Leaky ReLU).
●● Softmax - what does it do and why exponentials?
Answer: Turns logits into a probability distribution. Exponentials make
everything positive and disproportionately amplify the largest logit,
letting the distribution peak sharply around the most confident class.
●● Dropout - why does it help?
Answer: It forces units to learn independently useful features rather than
relying on co-adapted combinations. Like students in a group project: if
some randomly drop out, everyone else is forced to actually learn the
material.
REGULARIZATION COMPLETE EXAM
REVIEW WITH FULL SOLUTIONS
●● How do you distinguish underfitting from overfitting in practice?
Answer: Look at the gap between train and validation loss. Underfitting:
both are high and close together (model hasn't learned). Overfitting:
train loss is low but validation loss is high (model memorised noise).
●● What is regularisation actually doing?
Answer: It injects a prior to constrain the hypothesis space. It trades a
small amount of training fit for better generalisation, preventing the
model from chasing data quirks. L2 pushes for small weights, L1 for
sparsity, augmentation encodes invariances.
●● What does the optimiser landscape look like and why does SGD
work?
Answer: In high dimensions, it's dominated by saddle points, not local
minima; there's usually an escape route. SGD works because its noisy
mini-batch estimate acts as an implicit regulariser, biasing the optimiser
away from sharp minima and toward flat, wide ones that generalise
better.
●● Why do neural networks learn at all?
, Answer: Hierarchical feature learning (edges -> parts -> objects)
combined with backpropagation, which systematically distributes
credit/blame so every parameter gets a personalised, local update signal.
●● What's the real effect of batch size?
Answer: Larger batches give a lower-variance gradient estimate (faster
compute), but often generalise worse because they remove the implicit
regularisation of small-batch gradient noise, converging to sharper
minima.
●● What problem does ReLU solve and what is the 'dying' problem?
Answer: It solves the vanishing gradient problem of sigmoid/tanh by
keeping a constant gradient of 1 for positive inputs. The 'dying' problem
occurs if a large update pushes a unit permanently below zero, killing its
gradient forever (fixed by Leaky ReLU).
●● Softmax - what does it do and why exponentials?
Answer: Turns logits into a probability distribution. Exponentials make
everything positive and disproportionately amplify the largest logit,
letting the distribution peak sharply around the most confident class.
●● Dropout - why does it help?
Answer: It forces units to learn independently useful features rather than
relying on co-adapted combinations. Like students in a group project: if
some randomly drop out, everyone else is forced to actually learn the
material.
