CS 7643 Quiz 4 – Concepts Notes Update with complete solutions

Covers Structured Representations (Lesson 11), Language Models (Lesson 12), and Embeddings (Lesson 13). Conceptual Questions: • RNNs and LSTMs, how their update rules differ, and what problems they each have or solve RNN – Recurrent NNs, are designed to model sequences • Many to Many: Input a sequence - output a sequence ; aka sequence transduction. Other setups know as Encoder - Decoder OCR – given an image of a text, split that up into individual characters and try to recognize each one. • Many to one: sequence as input, one output Sentiment Analysis – given a piece of text, classify if the author was feeling positive or negative when writing. • One to many: one input, sequence as output (eg. Image captioning model) • One to one: no sequence involved, typical regression problems. RNNs solve the problems MLP (Multilayer Perceptron) have when used to model sequences. The functions f_theta is always the same – giving the recursive nature to the nn. Problem of RNN: Backprop gets complicated. The computational graph can be really long if the sequences are very long – updates are very expensive. Solution: ‘truncated backprop through time’ states are carried forward forever but just backprop for a fixed number of steps. Update rule for RNN Problem of the Vanilla RNN: Vanishing gradients Solution: LSTM architecture LSTM (Long Short-Term Memory) introduces the concept of gates – taking parts of the input to the cell, and multiply them together. Update rule for LSTM for c_t its update has an additive element to take care of the vanishing gradients problem.