CS 7641 RL MIDTERM PROBLEMS AND
SOLUTIONS EXAM QUESTIONS AND
CORRECT ANSWERS (VERIFIED ANSWERS)
PLUS RATIONALES 2026 Q&A | INSTANT
DOWNLOAD PDF.
CORE DOMAINS
Markov Decision Processes (MDPs)
Value Iteration and Policy Iteration
Q-Learning and Temporal Difference Learning
Exploration vs. Exploitation
Approximate Q-Learning and Feature Representation
Policy Evaluation and Policy Improvement
Convergence Properties in Reinforcement Learning
INTRODUCTION
The CS 7641 Reinforcement Learning Midterm Examination evaluates
graduate-level competency in the mathematical foundations and algorithmic
implementations of reinforcement learning. The assessment covers Markov
Decision Processes, dynamic programming approaches, temporal difference
learning, and function approximation techniques. Questions emphasize the
Bellman equations, convergence guarantees, and the distinction between on-
policy and off-policy methods. Emphasis is placed on deriving optimal policies,
computing Q-value updates, and understanding the theoretical underpinnings
that govern agent decision-making in sequential environments.
,SECTION ONE: QUESTIONS 1–100
1. In a Markov Decision Process, what does the Markov property state?
A. The reward function depends only on the current state
B. The next state depends only on the current state and action, not on history
C. All states are equally likely to be visited
D. The policy is always deterministic
B. The next state depends only on the current state and action, not on
history
RATIONALE: The Markov property states that the future is independent
of the past given the present. In an MDP, the transition probability P(s'|s,a)
depends only on the current state and action, not on the sequence of states
that preceded it.
2. Which of the following correctly defines the value function V^π(s) for
a policy π?
A. The immediate reward received in state s
B. The expected discounted sum of rewards starting from s and following π
C. The maximum reward achievable from state s
D. The probability of reaching state s under π
B. The expected discounted sum of rewards starting from s and following
π
, RATIONALE: The state-value function V^π(s) represents the expected
return (discounted sum of rewards) when starting in state s and following
policy π thereafter.
3. What is the Bellman optimality equation for V*(s)?
A. V(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V(s')]
B. V(s) = Σ_a π(a|s) [R(s,a) + γ V(s')]
C. V(s) = R(s) + γ min_a V(s')
D. V(s) = γ max_a V(s')
A. V(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V(s')]
RATIONALE: The Bellman optimality equation expresses the optimal
value of a state as the maximum over actions of the immediate reward plus
the discounted expected value of the next state, assuming optimal behavior
thereafter.
4. In value iteration, what is the update rule for V_{i+1}(s)?
A. V_{i+1}(s) = R(s) + γ V_i(s)
B. V_{i+1}(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V_i(s')]
C. V_{i+1}(s) = Σ_a π(a|s) V_i(s')
D. V_{i+1}(s) = V_i(s) + α [R(s) - V_i(s)]
B. V_{i+1}(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V_i(s')]
RATIONALE: Value iteration applies the Bellman optimality equation as
an iterative update, starting from arbitrary initial values and repeatedly
applying the backup operation until convergence.
, 5. What distinguishes policy iteration from value iteration?
A. Policy iteration alternates between policy evaluation and policy
improvement
B. Policy iteration does not require solving a system of equations
C. Policy iteration always converges in fewer iterations
D. Policy iteration only works for deterministic policies
A. Policy iteration alternates between policy evaluation and policy
improvement
RATIONALE: Policy iteration consists of two phases: policy evaluation
(computing V^π for the current policy) and policy improvement (updating
the policy greedily based on V^π). Value iteration combines these steps into a
single update.
6. In Q-learning, what is the update rule for Q(s,a)?
A. Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
B. Q(s,a) ← Q(s,a) + α[r + γ Q(s',a') - Q(s,a)]
C. Q(s,a) ← max_a' Q(s',a') + α[r - Q(s,a)]
D. Q(s,a) ← r + γ Q(s',a')
A. Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
RATIONALE: The Q-learning update uses the maximum Q-value of the
next state (off-policy) rather than the value of the action actually taken. The
learning rate α controls how much new information overrides old estimates.
7. What makes Q-learning an off-policy algorithm?
A. It requires the optimal policy to generate samples
SOLUTIONS EXAM QUESTIONS AND
CORRECT ANSWERS (VERIFIED ANSWERS)
PLUS RATIONALES 2026 Q&A | INSTANT
DOWNLOAD PDF.
CORE DOMAINS
Markov Decision Processes (MDPs)
Value Iteration and Policy Iteration
Q-Learning and Temporal Difference Learning
Exploration vs. Exploitation
Approximate Q-Learning and Feature Representation
Policy Evaluation and Policy Improvement
Convergence Properties in Reinforcement Learning
INTRODUCTION
The CS 7641 Reinforcement Learning Midterm Examination evaluates
graduate-level competency in the mathematical foundations and algorithmic
implementations of reinforcement learning. The assessment covers Markov
Decision Processes, dynamic programming approaches, temporal difference
learning, and function approximation techniques. Questions emphasize the
Bellman equations, convergence guarantees, and the distinction between on-
policy and off-policy methods. Emphasis is placed on deriving optimal policies,
computing Q-value updates, and understanding the theoretical underpinnings
that govern agent decision-making in sequential environments.
,SECTION ONE: QUESTIONS 1–100
1. In a Markov Decision Process, what does the Markov property state?
A. The reward function depends only on the current state
B. The next state depends only on the current state and action, not on history
C. All states are equally likely to be visited
D. The policy is always deterministic
B. The next state depends only on the current state and action, not on
history
RATIONALE: The Markov property states that the future is independent
of the past given the present. In an MDP, the transition probability P(s'|s,a)
depends only on the current state and action, not on the sequence of states
that preceded it.
2. Which of the following correctly defines the value function V^π(s) for
a policy π?
A. The immediate reward received in state s
B. The expected discounted sum of rewards starting from s and following π
C. The maximum reward achievable from state s
D. The probability of reaching state s under π
B. The expected discounted sum of rewards starting from s and following
π
, RATIONALE: The state-value function V^π(s) represents the expected
return (discounted sum of rewards) when starting in state s and following
policy π thereafter.
3. What is the Bellman optimality equation for V*(s)?
A. V(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V(s')]
B. V(s) = Σ_a π(a|s) [R(s,a) + γ V(s')]
C. V(s) = R(s) + γ min_a V(s')
D. V(s) = γ max_a V(s')
A. V(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V(s')]
RATIONALE: The Bellman optimality equation expresses the optimal
value of a state as the maximum over actions of the immediate reward plus
the discounted expected value of the next state, assuming optimal behavior
thereafter.
4. In value iteration, what is the update rule for V_{i+1}(s)?
A. V_{i+1}(s) = R(s) + γ V_i(s)
B. V_{i+1}(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V_i(s')]
C. V_{i+1}(s) = Σ_a π(a|s) V_i(s')
D. V_{i+1}(s) = V_i(s) + α [R(s) - V_i(s)]
B. V_{i+1}(s) = max_a [R(s,a) + γ Σ_s' P(s'|s,a) V_i(s')]
RATIONALE: Value iteration applies the Bellman optimality equation as
an iterative update, starting from arbitrary initial values and repeatedly
applying the backup operation until convergence.
, 5. What distinguishes policy iteration from value iteration?
A. Policy iteration alternates between policy evaluation and policy
improvement
B. Policy iteration does not require solving a system of equations
C. Policy iteration always converges in fewer iterations
D. Policy iteration only works for deterministic policies
A. Policy iteration alternates between policy evaluation and policy
improvement
RATIONALE: Policy iteration consists of two phases: policy evaluation
(computing V^π for the current policy) and policy improvement (updating
the policy greedily based on V^π). Value iteration combines these steps into a
single update.
6. In Q-learning, what is the update rule for Q(s,a)?
A. Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
B. Q(s,a) ← Q(s,a) + α[r + γ Q(s',a') - Q(s,a)]
C. Q(s,a) ← max_a' Q(s',a') + α[r - Q(s,a)]
D. Q(s,a) ← r + γ Q(s',a')
A. Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
RATIONALE: The Q-learning update uses the maximum Q-value of the
next state (off-policy) rather than the value of the action actually taken. The
learning rate α controls how much new information overrides old estimates.
7. What makes Q-learning an off-policy algorithm?
A. It requires the optimal policy to generate samples
