CS 7643 DEEP LEARNING
COMPREHENSIVE EXAM STUDY GUIDE
WITH COMPLETE SOLUTIONS
●● Definition: Machine_learning: Artificial_neural_network
Answer: Most modern deep learning models are based on artificial
neural networks, specifically convolutional neural networks (CNN)s,
although they can also include propositional formulas or latent variables
organized layer-wise in deep generative models such as the nodes in
deep belief networks and deep Boltzmann machines.
●● Definition: Artificial_neural_network: Matrix_(mathematics)
Answer: In deep learning, each level learns to transform its input data
into a slightly more abstract and composite representation. In an image
recognition application, the raw input may be a matrix of pixels; the first
representational layer may abstract the pixels and encode edges; the
second layer may compose and encode arrangements of edges; the third
layer may encode a nose and eyes; and the fourth layer may recognize
that the image contains a face. Importantly, a deep learning process can
learn which features to optimally place in which level on its own. This
does not eliminate the need for hand-tuning; for example, varying
numbers of layers and layer sizes can provide different degrees of
abstraction
●● Definition: Matrix_(mathematics): CAP
,Answer: The word ""deep"" in ""deep learning"" refers to the number of
layers through which the data is transformed. More precisely, deep
learning systems have a substantial credit assignment path (CAP) depth.
The CAP is the chain of transformations from input to output. CAPs
describe potentially causal connections between input and output. For a
feedforward neural network, the depth of the CAPs is that of the network
and is the number of hidden layers plus one (as the output layer is also
parameterized). For recurrent neural networks, in which a signal may
propagate through a layer more than once, the CAP depth is potentially
unlimited.[12] No universally agreed-upon threshold of depth divides
shallow learning from deep learning, but most researchers agree that
deep learning involves CAP depth higher than 2. CAP of depth 2 has
been shown to be a universal approximator in the sense that it can
emulate any function.[13] Beyond that, more layers do not add to the
function approximator ability of the network. Deep models (CAP > 2)
are able to extract better features than shallow models and hence, extra
layers help in learning the features effectively.
●● Definition: F neural_network: Greedy_algorithm
Answer: Deep learning architectures can be constructed with a greedy
layer-by-layer method. Deep learning helps to disentangle these
abstractions and pick out which features improve performance.
●● Definition: Greedy_algorithm: Supervised_learning
Answer: ['For supervised learning tasks, deep learning methods
eliminate feature engineering, by translating the data into compact
intermediate representations akin to principal components, and derive
layered structures that remove redundancy in representation.\n']
,●● Definition: Supervised_learning: Deep_belief_network
Answer: ['Deep learning algorithms can be applied to unsupervised
learning tasks. This is an important benefit because unlabeled data are
more abundant than the labeled data. Examples of deep structures that
can be trained in an unsupervised manner are deep belief
networks.[10][15]\n']
●● Definition: Deep_belief_networ: Universal_approximation_theorem
Answer: ['Deep neural networks are generally interpreted in terms of the
universal approximation theorem[16][17][18][19][20] or probabilistic
inference.[21][8][10][12][22]\n']
●● Definition: Universal_approximation_theorem:
Feedforward_neural_networks
Answer: ['The classic universal approximation theorem concerns the
capacity of feedforward neural networks with a single hidden layer of
finite size to approximate continuous functions.[16][17][18][19] In
1989, the first proof was published by George Cybenko for sigmoid
activation functions[16] and was generalised to feed-forward multi-layer
architectures in 1991 by Kurt Hornik.[17] Recent work also showed that
universal approximation also holds for non-bounded activation functions
such as the rectified linear unit.[23]\n']
●● Definition: Feedforward_neural_networks: Deep_neural_networ
, Answer: ['The universal approximation theorem for deep neural
networks concerns the capacity of networks with bounded width but the
depth is allowed to grow. Lu et al.[20] proved that if the width of a deep
neural network with ReLU activation is strictly larger than the input
dimension, then the network can approximate any Lebesgue integrable
function; If the width is smaller or equal to the input dimension, then a
deep neural network is not a universal approximator.\n']
●● Definition: Deep_neural_networ: Probabilistic
Answer: ['The probabilistic interpretation[22] derives from the field of
machine learning. It features inference,[8][9][10][12][15][22] as well as
the optimization concepts of training and testing, related to fitting and
generalization, respectively. More specifically, the probabilistic
interpretation considers the activation nonlinearity as a cumulative
distribution function.[22] The probabilistic interpretation led to the
introduction of dropout as regularizer in neural networks. The
probabilistic interpretation was introduced by researchers including
Hopfield, Widrow and Narendra and popularized in surveys such as the
one by Bishop.[24]\n']
●● Definition: Probabilistic: Frank_Rosenblatt
Answer: "['Some sources point out that Frank Rosenblatt developed and
explored all of the basic ingredients of the deep learning systems of
today.[25] He described it in his book ""Principles of Neurodynamics:
Perceptrons and the Theory of Brain Mechanisms"", published by
Cornell Aeronautical Laboratory, Inc., Cornell University in 1962.\n']"
COMPREHENSIVE EXAM STUDY GUIDE
WITH COMPLETE SOLUTIONS
●● Definition: Machine_learning: Artificial_neural_network
Answer: Most modern deep learning models are based on artificial
neural networks, specifically convolutional neural networks (CNN)s,
although they can also include propositional formulas or latent variables
organized layer-wise in deep generative models such as the nodes in
deep belief networks and deep Boltzmann machines.
●● Definition: Artificial_neural_network: Matrix_(mathematics)
Answer: In deep learning, each level learns to transform its input data
into a slightly more abstract and composite representation. In an image
recognition application, the raw input may be a matrix of pixels; the first
representational layer may abstract the pixels and encode edges; the
second layer may compose and encode arrangements of edges; the third
layer may encode a nose and eyes; and the fourth layer may recognize
that the image contains a face. Importantly, a deep learning process can
learn which features to optimally place in which level on its own. This
does not eliminate the need for hand-tuning; for example, varying
numbers of layers and layer sizes can provide different degrees of
abstraction
●● Definition: Matrix_(mathematics): CAP
,Answer: The word ""deep"" in ""deep learning"" refers to the number of
layers through which the data is transformed. More precisely, deep
learning systems have a substantial credit assignment path (CAP) depth.
The CAP is the chain of transformations from input to output. CAPs
describe potentially causal connections between input and output. For a
feedforward neural network, the depth of the CAPs is that of the network
and is the number of hidden layers plus one (as the output layer is also
parameterized). For recurrent neural networks, in which a signal may
propagate through a layer more than once, the CAP depth is potentially
unlimited.[12] No universally agreed-upon threshold of depth divides
shallow learning from deep learning, but most researchers agree that
deep learning involves CAP depth higher than 2. CAP of depth 2 has
been shown to be a universal approximator in the sense that it can
emulate any function.[13] Beyond that, more layers do not add to the
function approximator ability of the network. Deep models (CAP > 2)
are able to extract better features than shallow models and hence, extra
layers help in learning the features effectively.
●● Definition: F neural_network: Greedy_algorithm
Answer: Deep learning architectures can be constructed with a greedy
layer-by-layer method. Deep learning helps to disentangle these
abstractions and pick out which features improve performance.
●● Definition: Greedy_algorithm: Supervised_learning
Answer: ['For supervised learning tasks, deep learning methods
eliminate feature engineering, by translating the data into compact
intermediate representations akin to principal components, and derive
layered structures that remove redundancy in representation.\n']
,●● Definition: Supervised_learning: Deep_belief_network
Answer: ['Deep learning algorithms can be applied to unsupervised
learning tasks. This is an important benefit because unlabeled data are
more abundant than the labeled data. Examples of deep structures that
can be trained in an unsupervised manner are deep belief
networks.[10][15]\n']
●● Definition: Deep_belief_networ: Universal_approximation_theorem
Answer: ['Deep neural networks are generally interpreted in terms of the
universal approximation theorem[16][17][18][19][20] or probabilistic
inference.[21][8][10][12][22]\n']
●● Definition: Universal_approximation_theorem:
Feedforward_neural_networks
Answer: ['The classic universal approximation theorem concerns the
capacity of feedforward neural networks with a single hidden layer of
finite size to approximate continuous functions.[16][17][18][19] In
1989, the first proof was published by George Cybenko for sigmoid
activation functions[16] and was generalised to feed-forward multi-layer
architectures in 1991 by Kurt Hornik.[17] Recent work also showed that
universal approximation also holds for non-bounded activation functions
such as the rectified linear unit.[23]\n']
●● Definition: Feedforward_neural_networks: Deep_neural_networ
, Answer: ['The universal approximation theorem for deep neural
networks concerns the capacity of networks with bounded width but the
depth is allowed to grow. Lu et al.[20] proved that if the width of a deep
neural network with ReLU activation is strictly larger than the input
dimension, then the network can approximate any Lebesgue integrable
function; If the width is smaller or equal to the input dimension, then a
deep neural network is not a universal approximator.\n']
●● Definition: Deep_neural_networ: Probabilistic
Answer: ['The probabilistic interpretation[22] derives from the field of
machine learning. It features inference,[8][9][10][12][15][22] as well as
the optimization concepts of training and testing, related to fitting and
generalization, respectively. More specifically, the probabilistic
interpretation considers the activation nonlinearity as a cumulative
distribution function.[22] The probabilistic interpretation led to the
introduction of dropout as regularizer in neural networks. The
probabilistic interpretation was introduced by researchers including
Hopfield, Widrow and Narendra and popularized in surveys such as the
one by Bishop.[24]\n']
●● Definition: Probabilistic: Frank_Rosenblatt
Answer: "['Some sources point out that Frank Rosenblatt developed and
explored all of the basic ingredients of the deep learning systems of
today.[25] He described it in his book ""Principles of Neurodynamics:
Perceptrons and the Theory of Brain Mechanisms"", published by
Cornell Aeronautical Laboratory, Inc., Cornell University in 1962.\n']"
