DEEP LEARNING OPTIMIZATION AND
REGULARIZATION COMPREHENSIVE
EXAM STUDY GUIDE WITH COMPLETE
SOLUTIONS
●● What is the difference between backpropagation and optimisation?
Answer: Backpropagation calculates the gradients of the cost with
respect to every parameter. The optimiser uses those gradients to update
the parameters.
●● What is the standard gradient-descent update rule?
Answer: w ← w - η ∂J/∂w, where w is a parameter, J is the cost and η is
the learning rate.
●● Why is the gradient subtracted in gradient descent?
Answer: The gradient points in the direction of the steepest increase in
cost. Subtracting it moves the parameters towards decreasing cost.
●● What does the magnitude of a gradient indicate?
Answer: It indicates how sensitive the cost is to a change in that
parameter. A large magnitude suggests a small parameter change may
strongly affect the cost.
,●● What does the sign of a gradient indicate?
Answer: It determines the direction in which the parameter should move
to reduce the cost.
●● What is a parameter? Give examples.
Answer: A parameter is learned directly during training. Examples
include weights and biases.
●● What is a hyperparameter? Give examples.
Answer: A hyperparameter is chosen before or around training rather
than learned through backpropagation. Examples include learning rate,
batch size, optimiser, momentum coefficient and network depth.
●● What is an epoch?
Answer: One complete pass through the entire training dataset.
●● What is an iteration?
Answer: One parameter update. An epoch may contain multiple
iterations when mini-batches or individual examples are used.
●● How many updates occur per epoch if a dataset contains 1,000
examples and the batch size is 100?
Answer: 10 iterations or parameter updates occur per epoch.
, ●● What is Batch Gradient Descent?
Answer: Batch GD calculates the gradient using the entire training
dataset before making one parameter update.
●● How many updates does Batch GD normally make per epoch?
Answer: One update per epoch.
●● What are the advantages of Batch GD?
Answer: It produces stable, accurate gradient estimates and usually gives
a smooth cost curve.
●● What are the disadvantages of Batch GD?
Answer: Each update can be slow, computationally expensive and
memory-intensive for large datasets.
●● What is Stochastic Gradient Descent?
Answer: SGD calculates the gradient from one training example and
immediately updates the parameters.
●● What are the advantages of SGD?
Answer: It produces frequent updates, can begin learning immediately
and introduces noise that may help it move through saddle points and
shallow local minima.
REGULARIZATION COMPREHENSIVE
EXAM STUDY GUIDE WITH COMPLETE
SOLUTIONS
●● What is the difference between backpropagation and optimisation?
Answer: Backpropagation calculates the gradients of the cost with
respect to every parameter. The optimiser uses those gradients to update
the parameters.
●● What is the standard gradient-descent update rule?
Answer: w ← w - η ∂J/∂w, where w is a parameter, J is the cost and η is
the learning rate.
●● Why is the gradient subtracted in gradient descent?
Answer: The gradient points in the direction of the steepest increase in
cost. Subtracting it moves the parameters towards decreasing cost.
●● What does the magnitude of a gradient indicate?
Answer: It indicates how sensitive the cost is to a change in that
parameter. A large magnitude suggests a small parameter change may
strongly affect the cost.
,●● What does the sign of a gradient indicate?
Answer: It determines the direction in which the parameter should move
to reduce the cost.
●● What is a parameter? Give examples.
Answer: A parameter is learned directly during training. Examples
include weights and biases.
●● What is a hyperparameter? Give examples.
Answer: A hyperparameter is chosen before or around training rather
than learned through backpropagation. Examples include learning rate,
batch size, optimiser, momentum coefficient and network depth.
●● What is an epoch?
Answer: One complete pass through the entire training dataset.
●● What is an iteration?
Answer: One parameter update. An epoch may contain multiple
iterations when mini-batches or individual examples are used.
●● How many updates occur per epoch if a dataset contains 1,000
examples and the batch size is 100?
Answer: 10 iterations or parameter updates occur per epoch.
, ●● What is Batch Gradient Descent?
Answer: Batch GD calculates the gradient using the entire training
dataset before making one parameter update.
●● How many updates does Batch GD normally make per epoch?
Answer: One update per epoch.
●● What are the advantages of Batch GD?
Answer: It produces stable, accurate gradient estimates and usually gives
a smooth cost curve.
●● What are the disadvantages of Batch GD?
Answer: Each update can be slow, computationally expensive and
memory-intensive for large datasets.
●● What is Stochastic Gradient Descent?
Answer: SGD calculates the gradient from one training example and
immediately updates the parameters.
●● What are the advantages of SGD?
Answer: It produces frequent updates, can begin learning immediately
and introduces noise that may help it move through saddle points and
shallow local minima.
