DEEP LEARNING OPTIMIZATION AND
REGULARIZATION EXAM PREPARATION
TEST BANK WITH CORRECT ANSWERS
●● representational capacity
Answer: the family of functions used by the learning
algorithm
§ Polynomial vs. neural network regression
●● effective capacity
Answer: Imperfection of the optimization algorithm
can be less than the representational capacity
●● Occam's razor (principle of parsimony)
Answer: Among competing
hypotheses that explain observations equal well, we should
choose the "simplest" one
●● Vapnic-Chervonenkis (VC) dimension
,Answer: measures the capacity of a
model f as the maximum number of points that can be labeled
arbitrarily (basically given a paramset theta, how PERFECTLY can it
label things)
●● Bayes error:
Answer: Error incurred by an oracle than know the true data generating
distribution (because of noise)
●● Regularization
Answer: any modification we make to
a learning algorithm to reduce its
generalization error but not its training error
●● types of regularization
Answer: § 𝐿7 parameter regularization
§ 𝐿8 parameter regularization
§ Data set augmentation
§ Injecting noise
§ Early stopping
§ Bagging (bootstrap aggregating)
§ Dropout
, ●● k-fold Cross Validation
Answer: § Partition the dataset into k nonoverlapping subsets
§ The test error is computed by averaging the test error across k
trials
§ On trial i, the i-th subset is used as the test set and the rest of the
data is used as the training set
●● stochastic gradient descent
Answer: SGD approximates the gradient using a small set of examples
(minibatch)
●● Most popular type of Grad. Descent for DL models?
Answer: Stochastic
●● DL motivations
Answer: § Curse of dimensionality: Machine learning algorithms
become exceedingly difficult when the number of
dimensions of the data is high
§ Deep learning algorithms exhibit reduced
generalization error in high-dimensions
REGULARIZATION EXAM PREPARATION
TEST BANK WITH CORRECT ANSWERS
●● representational capacity
Answer: the family of functions used by the learning
algorithm
§ Polynomial vs. neural network regression
●● effective capacity
Answer: Imperfection of the optimization algorithm
can be less than the representational capacity
●● Occam's razor (principle of parsimony)
Answer: Among competing
hypotheses that explain observations equal well, we should
choose the "simplest" one
●● Vapnic-Chervonenkis (VC) dimension
,Answer: measures the capacity of a
model f as the maximum number of points that can be labeled
arbitrarily (basically given a paramset theta, how PERFECTLY can it
label things)
●● Bayes error:
Answer: Error incurred by an oracle than know the true data generating
distribution (because of noise)
●● Regularization
Answer: any modification we make to
a learning algorithm to reduce its
generalization error but not its training error
●● types of regularization
Answer: § 𝐿7 parameter regularization
§ 𝐿8 parameter regularization
§ Data set augmentation
§ Injecting noise
§ Early stopping
§ Bagging (bootstrap aggregating)
§ Dropout
, ●● k-fold Cross Validation
Answer: § Partition the dataset into k nonoverlapping subsets
§ The test error is computed by averaging the test error across k
trials
§ On trial i, the i-th subset is used as the test set and the rest of the
data is used as the training set
●● stochastic gradient descent
Answer: SGD approximates the gradient using a small set of examples
(minibatch)
●● Most popular type of Grad. Descent for DL models?
Answer: Stochastic
●● DL motivations
Answer: § Curse of dimensionality: Machine learning algorithms
become exceedingly difficult when the number of
dimensions of the data is high
§ Deep learning algorithms exhibit reduced
generalization error in high-dimensions
