CS7643 Last Quiz – 2025/2026 Updated Study
Guide & Practice Questions | 100% Pass
Guarantee
Reinforcement learning - correct answerSequential decision making in an environment
with evaluative feedback
Environment: may be unknown, non-linear, stochastic and complex
Agent: learns a policy to map states of the environments to actions
- seeks to maximize long-term reward
RL: Evaluative Feedback - correct answer- Pick an action, receive a reward
- No supervision for what the correct action is or would have been (unlike supervised
learning)
RL: Sequential Decisions - correct answer- Plan and execution actions over a sequence
of states
- Reward may be delayed, requiring optimization of future rewards (long-term planning)
Signature Challenges in RL - correct answerEvaluative Feedback: Need trial and error
to find the right action
Delayed Feedback: Actions may not lead to immediate reward
Non-stationarity: Data distribution of visited states changes when the policy changes
Fleeting Nature: of online data (may only see data once)
MDP - correct answerFramework underlying RL
S: Set of states
A: Set of actions
R: Distribution of Rewards
T: Transition probabiliity
y: Discount property
Markov Property: Current state completely characterizes state of the environment
RL: Equations relating optimal quantities - correct answer1. V*(S) = max_a(Q*(s, a)
2. PI*(s) = argmax_a(Q*(s, a)
V*(S) - correct answermax_a (sum_(s') { p(s'|s, a) [r(s, a) + yV*(s')] } )
, Q*(s,a) - correct answersum_(s') { p(s'|s, a) [r(s, a) + y*max_(a'){Q*(s', a') ] }
Value Iteration - correct answerv_(i+1) = max_a (sum_(s') { p(s'|s, a) [r(s, a) + yV_(i)(s')]
})
- repeat until convergence
- Time complexity per iteration O(|S^2| |A|)
Policy Iteration - correct answerPolicy Evaluation: Compute V(pi)
Policy Refinement: Greedily change action as per V(Pi) at next states
Why do Policy Iteration: PI_i often converges to PI* sooner than V_PI to V_PI*
- thus requires few iterations
Deep Q-Learning - correct answer- Q(s, a; w, b) = w_a^t * s + b_a
MSE Loss := (Q_new(s, a) - (r + y*max_a(Q_old(s', a)))^2
- using a single Q function makes loss function unstable
--> use two Q-tables (NNs)
- Freeze Q_old and update Q_new
- Set Q_old = Q_new at regular intervals
Fitted Q-Iteration - correct answerAlgorithm to optimize MSE Loss on a fixed dataset
RL: How to Collect Data - correct answerChallenge 1: Exploration vs Exploitation
Challenge 2: Non iid, highly correlated data
- This leads to high variance in gradients and inefficient learning
- Experience Replay Addresses this:
--> store (s, a, s', r) pairs and continually update episodes (older samples discarded)
--> Train Q-Network on random mini batches of transitions from the replay memory
instead of consecutive examples
--> larger the buffer, lower the correlation
Experience Replay - correct answer- store (s, a, s', r) pairs and continually update
episodes (older samples discarded)
- Train Q-Network on random mini batches of transitions from the replay memory
instead of consecutive examples
- larger the buffer, lower the correlation
Value Based RL Methods Learn, Model Based Methods Learn - correct answer- Q-
Functions
Guide & Practice Questions | 100% Pass
Guarantee
Reinforcement learning - correct answerSequential decision making in an environment
with evaluative feedback
Environment: may be unknown, non-linear, stochastic and complex
Agent: learns a policy to map states of the environments to actions
- seeks to maximize long-term reward
RL: Evaluative Feedback - correct answer- Pick an action, receive a reward
- No supervision for what the correct action is or would have been (unlike supervised
learning)
RL: Sequential Decisions - correct answer- Plan and execution actions over a sequence
of states
- Reward may be delayed, requiring optimization of future rewards (long-term planning)
Signature Challenges in RL - correct answerEvaluative Feedback: Need trial and error
to find the right action
Delayed Feedback: Actions may not lead to immediate reward
Non-stationarity: Data distribution of visited states changes when the policy changes
Fleeting Nature: of online data (may only see data once)
MDP - correct answerFramework underlying RL
S: Set of states
A: Set of actions
R: Distribution of Rewards
T: Transition probabiliity
y: Discount property
Markov Property: Current state completely characterizes state of the environment
RL: Equations relating optimal quantities - correct answer1. V*(S) = max_a(Q*(s, a)
2. PI*(s) = argmax_a(Q*(s, a)
V*(S) - correct answermax_a (sum_(s') { p(s'|s, a) [r(s, a) + yV*(s')] } )
, Q*(s,a) - correct answersum_(s') { p(s'|s, a) [r(s, a) + y*max_(a'){Q*(s', a') ] }
Value Iteration - correct answerv_(i+1) = max_a (sum_(s') { p(s'|s, a) [r(s, a) + yV_(i)(s')]
})
- repeat until convergence
- Time complexity per iteration O(|S^2| |A|)
Policy Iteration - correct answerPolicy Evaluation: Compute V(pi)
Policy Refinement: Greedily change action as per V(Pi) at next states
Why do Policy Iteration: PI_i often converges to PI* sooner than V_PI to V_PI*
- thus requires few iterations
Deep Q-Learning - correct answer- Q(s, a; w, b) = w_a^t * s + b_a
MSE Loss := (Q_new(s, a) - (r + y*max_a(Q_old(s', a)))^2
- using a single Q function makes loss function unstable
--> use two Q-tables (NNs)
- Freeze Q_old and update Q_new
- Set Q_old = Q_new at regular intervals
Fitted Q-Iteration - correct answerAlgorithm to optimize MSE Loss on a fixed dataset
RL: How to Collect Data - correct answerChallenge 1: Exploration vs Exploitation
Challenge 2: Non iid, highly correlated data
- This leads to high variance in gradients and inefficient learning
- Experience Replay Addresses this:
--> store (s, a, s', r) pairs and continually update episodes (older samples discarded)
--> Train Q-Network on random mini batches of transitions from the replay memory
instead of consecutive examples
--> larger the buffer, lower the correlation
Experience Replay - correct answer- store (s, a, s', r) pairs and continually update
episodes (older samples discarded)
- Train Q-Network on random mini batches of transitions from the replay memory
instead of consecutive examples
- larger the buffer, lower the correlation
Value Based RL Methods Learn, Model Based Methods Learn - correct answer- Q-
Functions
