CS7643 QUIZ 5 EXAM | QUESTIONS AND ANSWERS | VERIFIED AND
WELL DETAILED ANSWERS | LATEST EXAM UPDATE
Properties of the softmax function
1.) Softmax is permutation equivariant (permutation of input leads to the
same permutation of output)
2.) Softmax is not linear
3.) Softmax is differentiable
4.) Softmax interpolates between a distribution that selects index uniformly
at random and dist that selects the argmax index with probability 1.
Attention
weighting or probability distribution over inputs that depends on
computational state and inputs
what does softmax allow?
allows random sampling of numbers from original set with a distribution
that depends on the values from the numbers of the original set.
Softmax Attention
- Given a set. of vectors "U" and a query vector "Q"
- We can select the most similar vector to "Q" via p=softmax(U * Q)
what does Softmax attention do?
Gives a method of differentially selecting a vector from a set.
Forms of attention
hard and soft
Hard Attention
samples are drawn from the distribution over input
Soft Attention
summary over inputs by using distribution as weighted average.
The encoder-decoder model and how it's used by recurrent neural
networks for seq2seq prediction problems.
,1.) Encoder RNN produces an encoding of source sentence -> passed to
decoder RNN
2.) Decoder RNN is a language model that generates target sentence,
conditioned on encoding
Beam search algorithm and how it works in seq2seq models using condition
probability.
-An algorithm that approximates the argmax for machine translation. The
argmax needs to be approximated because the exponential space is
intractable. However, this algorithm is able to search exponential space in
linear time.
-Beam size "k" determines the width of the search
How is machine translation modeled?
Translation is often modeled as a conditional language model.
Transfomers are known for?
conditional language models
Attention Function
-maps a query and a set of key-value pairs to an output, where the query,
keys, values, and output are all vectors
-The output is computed as a weighted sum of the values, where the weight
assigned to each value is computed by a compatibility function of the query
with the corresponding key.
Per-layer computational complexity
a self-attention layer connects all positions with a constant number of
sequentially executed operations, whereas a recurrent layer requires O(n)
sequential operations. In terms of computational complexity, self-attention
layers are faster than recurrent layers when the sequence length n is
smaller than the representation dimensionality d
transformer
an attention based, encoder-decoder type architecture.
Encoder Layer
- maps an input sequence into an abstract continuous representation that
holds all the learned information of that input.
- It contains 2 sub-modules, multi-headed attention, followed by a fully
, connected network.
- There are also residual connections around each of the two sublayers
followed by a layer normalization.
Decoder
- takes output from encoder as input and uses and step by step generates a
single output while also being fed the previous output.
- The decoder has a similar sub-layer as the encoder. it has two multi-
headed attention layers, a pointwise feed-forward layer, and residual
connections, and layer normalization after each sub-layer.
- The decoder is capped off with a linear layer that acts as a classifier, and a
softmax to get the word probabilities.
Encoder Word Embeddings
lookup table that maps a to its learned vector representation.
Encoder Positional Encoding
- inject positional information into the word embeddings.
- For every odd index on the input vector, create a vector using the cos
function.
- For every even index, create a vector using the sin function.
Encoder Multi-Headed Attention
- Multi-headed attention in the encoder applies a specific attention
mechanism called self-attention. Self-attention allows the models to
associate each word in the input, to other words.
Decoder First Multi-Headed Attention
- computes the attention scores for the decoder's input.
- Uses look ahead mask to prevent decoder from looking at future tokens
Decoder Second Multi-Headed Attention
- takes the query and keys from the encoder as well as the output from the
first decoder
- matches the encoder's input to the decoder's input, allowing the decoder
to decide which encoder input is relevant to put a focus on.
Reinforcement learning
Sequential decision making in an environment with evaluative feedback
WELL DETAILED ANSWERS | LATEST EXAM UPDATE
Properties of the softmax function
1.) Softmax is permutation equivariant (permutation of input leads to the
same permutation of output)
2.) Softmax is not linear
3.) Softmax is differentiable
4.) Softmax interpolates between a distribution that selects index uniformly
at random and dist that selects the argmax index with probability 1.
Attention
weighting or probability distribution over inputs that depends on
computational state and inputs
what does softmax allow?
allows random sampling of numbers from original set with a distribution
that depends on the values from the numbers of the original set.
Softmax Attention
- Given a set. of vectors "U" and a query vector "Q"
- We can select the most similar vector to "Q" via p=softmax(U * Q)
what does Softmax attention do?
Gives a method of differentially selecting a vector from a set.
Forms of attention
hard and soft
Hard Attention
samples are drawn from the distribution over input
Soft Attention
summary over inputs by using distribution as weighted average.
The encoder-decoder model and how it's used by recurrent neural
networks for seq2seq prediction problems.
,1.) Encoder RNN produces an encoding of source sentence -> passed to
decoder RNN
2.) Decoder RNN is a language model that generates target sentence,
conditioned on encoding
Beam search algorithm and how it works in seq2seq models using condition
probability.
-An algorithm that approximates the argmax for machine translation. The
argmax needs to be approximated because the exponential space is
intractable. However, this algorithm is able to search exponential space in
linear time.
-Beam size "k" determines the width of the search
How is machine translation modeled?
Translation is often modeled as a conditional language model.
Transfomers are known for?
conditional language models
Attention Function
-maps a query and a set of key-value pairs to an output, where the query,
keys, values, and output are all vectors
-The output is computed as a weighted sum of the values, where the weight
assigned to each value is computed by a compatibility function of the query
with the corresponding key.
Per-layer computational complexity
a self-attention layer connects all positions with a constant number of
sequentially executed operations, whereas a recurrent layer requires O(n)
sequential operations. In terms of computational complexity, self-attention
layers are faster than recurrent layers when the sequence length n is
smaller than the representation dimensionality d
transformer
an attention based, encoder-decoder type architecture.
Encoder Layer
- maps an input sequence into an abstract continuous representation that
holds all the learned information of that input.
- It contains 2 sub-modules, multi-headed attention, followed by a fully
, connected network.
- There are also residual connections around each of the two sublayers
followed by a layer normalization.
Decoder
- takes output from encoder as input and uses and step by step generates a
single output while also being fed the previous output.
- The decoder has a similar sub-layer as the encoder. it has two multi-
headed attention layers, a pointwise feed-forward layer, and residual
connections, and layer normalization after each sub-layer.
- The decoder is capped off with a linear layer that acts as a classifier, and a
softmax to get the word probabilities.
Encoder Word Embeddings
lookup table that maps a to its learned vector representation.
Encoder Positional Encoding
- inject positional information into the word embeddings.
- For every odd index on the input vector, create a vector using the cos
function.
- For every even index, create a vector using the sin function.
Encoder Multi-Headed Attention
- Multi-headed attention in the encoder applies a specific attention
mechanism called self-attention. Self-attention allows the models to
associate each word in the input, to other words.
Decoder First Multi-Headed Attention
- computes the attention scores for the decoder's input.
- Uses look ahead mask to prevent decoder from looking at future tokens
Decoder Second Multi-Headed Attention
- takes the query and keys from the encoder as well as the output from the
first decoder
- matches the encoder's input to the decoder's input, allowing the decoder
to decide which encoder input is relevant to put a focus on.
Reinforcement learning
Sequential decision making in an environment with evaluative feedback
