Advanced Data Analytics WGU D204
EXAM BEST LATEST STUDYING
MATERIAL GUARANTEED PASS.
When should a Gradient Descent optimization algorithm be used?
- ANSWER - convex, smooth loss function
- small dataset/model
When should a Stochastic Gradient Descent optimization algorithm be used? -
ANSWER - 1 sample: want fast results, have low memory requirements
- mini-batch of samples: optimal subset that fits the memory
What does "momentum" do in optimizers (typically SGD)?
- ANSWER - accelerate the convergence of the optimization
- adds a fraction of the previous update vector to the current update
- smooth out fluctuations in the gradient
What is Adagrad?
- ANSWER - adaptively adjust the learning rates of individual parameters based on their
historical gradients
- learning rate is reduced if the gradient is very large
- different learning rate for each parameter
- eliminates the need to manually tune the learning rate
Problems with Adagrad
- ANSWER - accumulation of the squared gradients
- learning rate is always decreasing
What is AdaDelta
- ANSWER - restricts the gradient history used for parameter changing to a specific
size
- no need to set the initial learning rate
What is RMSProp?
- ANSWER - uses the idea of momentum
- also tries to solve the issues with Adagrad
- normalizes learning rate by the magnitudes of recent gradient of a weight
What is the Adam optimizer?
- ANSWER - keeps separate learning rates for each parameter
- keeps track of exponentially decaying average of past gradients
- combines RMSProp and Momentum
What are the 3 properties of the image classification task?
,- ANSWER 1. Some patterns are much smaller than the whole image (one neuron does
not need to see the whole image).
2. The same patterns appear in different regions.
3. Subsampling the pixels will not change the object.
What are filters?
- ANSWER - a.k.a "kernels"
- small, learnable matrices applied to the input data (e.g., images) through convolution
operations
- output of convolutional operation is a feature map
What is stride?
- ANSWER the size of the step the convolution filter moves each time
Explain "shared weights" in convolutional layers
- ANSWER - the same filter weights are applied to the entire spatial feature map input
- NOT the biases
- this reduces the number of free parameters
- same features can be extracted at every location
What problem is there with corner pixels in convolutional layers?
- ANSWER - do not get passed over as frequently as middle pixels
- less strength in detecting features in corners/edges
- solution: padding
What is dilation in convolutional layers?
- ANSWER - the spacing between the kernel elements
- increases the receptive field w/out increasing the number of parameters
What does the pooling layer do in convolutional neural networks?
- ANSWER - reduce the spatial dimension of each feature map
- certain degree of shift and distortion invariance achieved
- ex. Max Pooling
What does flattening do in CNNs? - ANSWER - process of converting the multi-
dimensional feature maps produced by convolutional and pooling layers into a one-
dimensional vector
What is "Bag of Words" in textual data analysis?
- ANSWER - technique for natural language processing that does not consider the
order of words
- focus is more on the frequency of occurrences
- uses one-hot encoding
- sparse data, no word similarity (i.e. woman and girl)
What is word embedding?
, - ANSWER - learning a lower-dimensional embedding for each input word
- able to measure the similarity between words
- similar words will have similar vector representations
What is a RNN?
- ANSWER - Recurrent Neural Network
- uses a hidden state to save information about the sequence that has been processed
so far
- produce an output at each time stamp
- able to handle variable-length sequences
- track long-term dependencies
- maintain information about order
- share parameters across the sequence
What is a bidirectional RNN?
- ANSWER - RNN that processes info from front to back and back to front
- hidden states for both the forward and backward passes are combined
- combined hidden state is used to generate output sequence
What is a common issue with RNNs?
- ANSWER Vanishing gradients: the contribution from earlier steps becomes
insignificant
What is LSTM?
- ANSWER - type of RNN that addresses the vanishing gradient problem
- gates control the information flow
- uses gating mechanism to remember information over long sequences
What is a GRU?
- ANSWER - Gated Recurrent Unit
- similar to LSTM; addresses vanishing gradient problem, captures long-range
dependencies
- use update and reset gates to control the flow of information
- have less parameters compared to LSTM, making them typically more efficient
Formula for the number of parameters in a convolutional layer (square kernel) -
ANSWER P = K^2 x C_in x C_out + C_out
Formula for the width (height) of the output matrix of a convolutional layer (square
kernel of size k x k) - ANSWER floor((W_in + 2P - k)/s) + 1
Which arrangement leads to better long-term memory in LSTM, stacked or nested? -
ANSWER Nested
What is a diffusion model? How does it differ from a GAN?
- ANSWER - like GANs, diffusion models are generative models
EXAM BEST LATEST STUDYING
MATERIAL GUARANTEED PASS.
When should a Gradient Descent optimization algorithm be used?
- ANSWER - convex, smooth loss function
- small dataset/model
When should a Stochastic Gradient Descent optimization algorithm be used? -
ANSWER - 1 sample: want fast results, have low memory requirements
- mini-batch of samples: optimal subset that fits the memory
What does "momentum" do in optimizers (typically SGD)?
- ANSWER - accelerate the convergence of the optimization
- adds a fraction of the previous update vector to the current update
- smooth out fluctuations in the gradient
What is Adagrad?
- ANSWER - adaptively adjust the learning rates of individual parameters based on their
historical gradients
- learning rate is reduced if the gradient is very large
- different learning rate for each parameter
- eliminates the need to manually tune the learning rate
Problems with Adagrad
- ANSWER - accumulation of the squared gradients
- learning rate is always decreasing
What is AdaDelta
- ANSWER - restricts the gradient history used for parameter changing to a specific
size
- no need to set the initial learning rate
What is RMSProp?
- ANSWER - uses the idea of momentum
- also tries to solve the issues with Adagrad
- normalizes learning rate by the magnitudes of recent gradient of a weight
What is the Adam optimizer?
- ANSWER - keeps separate learning rates for each parameter
- keeps track of exponentially decaying average of past gradients
- combines RMSProp and Momentum
What are the 3 properties of the image classification task?
,- ANSWER 1. Some patterns are much smaller than the whole image (one neuron does
not need to see the whole image).
2. The same patterns appear in different regions.
3. Subsampling the pixels will not change the object.
What are filters?
- ANSWER - a.k.a "kernels"
- small, learnable matrices applied to the input data (e.g., images) through convolution
operations
- output of convolutional operation is a feature map
What is stride?
- ANSWER the size of the step the convolution filter moves each time
Explain "shared weights" in convolutional layers
- ANSWER - the same filter weights are applied to the entire spatial feature map input
- NOT the biases
- this reduces the number of free parameters
- same features can be extracted at every location
What problem is there with corner pixels in convolutional layers?
- ANSWER - do not get passed over as frequently as middle pixels
- less strength in detecting features in corners/edges
- solution: padding
What is dilation in convolutional layers?
- ANSWER - the spacing between the kernel elements
- increases the receptive field w/out increasing the number of parameters
What does the pooling layer do in convolutional neural networks?
- ANSWER - reduce the spatial dimension of each feature map
- certain degree of shift and distortion invariance achieved
- ex. Max Pooling
What does flattening do in CNNs? - ANSWER - process of converting the multi-
dimensional feature maps produced by convolutional and pooling layers into a one-
dimensional vector
What is "Bag of Words" in textual data analysis?
- ANSWER - technique for natural language processing that does not consider the
order of words
- focus is more on the frequency of occurrences
- uses one-hot encoding
- sparse data, no word similarity (i.e. woman and girl)
What is word embedding?
, - ANSWER - learning a lower-dimensional embedding for each input word
- able to measure the similarity between words
- similar words will have similar vector representations
What is a RNN?
- ANSWER - Recurrent Neural Network
- uses a hidden state to save information about the sequence that has been processed
so far
- produce an output at each time stamp
- able to handle variable-length sequences
- track long-term dependencies
- maintain information about order
- share parameters across the sequence
What is a bidirectional RNN?
- ANSWER - RNN that processes info from front to back and back to front
- hidden states for both the forward and backward passes are combined
- combined hidden state is used to generate output sequence
What is a common issue with RNNs?
- ANSWER Vanishing gradients: the contribution from earlier steps becomes
insignificant
What is LSTM?
- ANSWER - type of RNN that addresses the vanishing gradient problem
- gates control the information flow
- uses gating mechanism to remember information over long sequences
What is a GRU?
- ANSWER - Gated Recurrent Unit
- similar to LSTM; addresses vanishing gradient problem, captures long-range
dependencies
- use update and reset gates to control the flow of information
- have less parameters compared to LSTM, making them typically more efficient
Formula for the number of parameters in a convolutional layer (square kernel) -
ANSWER P = K^2 x C_in x C_out + C_out
Formula for the width (height) of the output matrix of a convolutional layer (square
kernel of size k x k) - ANSWER floor((W_in + 2P - k)/s) + 1
Which arrangement leads to better long-term memory in LSTM, stacked or nested? -
ANSWER Nested
What is a diffusion model? How does it differ from a GAN?
- ANSWER - like GANs, diffusion models are generative models
