CS 7643 Quiz 5 Notes|2025 Update with complete solution.
Here are the "what's important points" for quiz 5 which covers Neural Attention Models
(Lesson 14), Neural Machine Translation (Lesson 15), and Advanced Topics (Lesson
16). /Special mention to "Attention is all you need".
● The encoder-decoder model and how it's used by recurrent neural networks for
seq2seq prediction problems.
● Beam search.
● Neural attention (different types, how its computed)
●
o “Softmax attention” lecture 14
o
Public
, o “Soft” attention calculation:
o Multilayer “soft” attention
● Transformers (architecture details, positional encoding, computational
complexity)
● Byte pair encoding.
Sequence 2 sequence models:
- https://www.youtube.com/watch?
v=_i3aqgKVNQI&list=PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6
- https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html
-
Public
, -
- Encoder (x network in image above) takes in French sentence
- Decoder (y network in image above) outputs English sentence
- Ex: Image captioning
o Use CNN to encodes an image
o Feed encoding to RNN to generate the caption
Most likely sentence/machine translation model:
- https://www.youtube.com/watch?
v=Er2ucMxjdHE&list=PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6&index=2
- Language model selects most likely sentence
- Decoder network looks similar to language model, except instead of starting with a vector of all
0s, it has an Encoder network that figures out a representation of the sentence and gives that to
the Decoder network
o Machine translation model = “conditional language model” models probability of
output English sentence conditioned on input French sentence
▪ Do not want to sample outputs at random; want to find the output that
maximizes the probability
● Beam search is the algorithm that maximizes the output probability
based on the input sentence
o (Not optimal to maximize one word at a time)
Beam search:
- C5W3L03 Beam Search
- Heuristic search algorithm; doesn’t always output most likely sentence
- Goal is to output the best/most likely translation, etc, not a random one
- Step 1: Beam search tries to pick the first English word (from the French translation) from a
vocabulary
Public
Here are the "what's important points" for quiz 5 which covers Neural Attention Models
(Lesson 14), Neural Machine Translation (Lesson 15), and Advanced Topics (Lesson
16). /Special mention to "Attention is all you need".
● The encoder-decoder model and how it's used by recurrent neural networks for
seq2seq prediction problems.
● Beam search.
● Neural attention (different types, how its computed)
●
o “Softmax attention” lecture 14
o
Public
, o “Soft” attention calculation:
o Multilayer “soft” attention
● Transformers (architecture details, positional encoding, computational
complexity)
● Byte pair encoding.
Sequence 2 sequence models:
- https://www.youtube.com/watch?
v=_i3aqgKVNQI&list=PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6
- https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html
-
Public
, -
- Encoder (x network in image above) takes in French sentence
- Decoder (y network in image above) outputs English sentence
- Ex: Image captioning
o Use CNN to encodes an image
o Feed encoding to RNN to generate the caption
Most likely sentence/machine translation model:
- https://www.youtube.com/watch?
v=Er2ucMxjdHE&list=PLkDaE6sCZn6F6wUI9tvS_Gw1vaFAx6rd6&index=2
- Language model selects most likely sentence
- Decoder network looks similar to language model, except instead of starting with a vector of all
0s, it has an Encoder network that figures out a representation of the sentence and gives that to
the Decoder network
o Machine translation model = “conditional language model” models probability of
output English sentence conditioned on input French sentence
▪ Do not want to sample outputs at random; want to find the output that
maximizes the probability
● Beam search is the algorithm that maximizes the output probability
based on the input sentence
o (Not optimal to maximize one word at a time)
Beam search:
- C5W3L03 Beam Search
- Heuristic search algorithm; doesn’t always output most likely sentence
- Goal is to output the best/most likely translation, etc, not a random one
- Step 1: Beam search tries to pick the first English word (from the French translation) from a
vocabulary
Public
