CS 7641 REINFORCEMENT LEARNING
PROBLEMS AND SOLUTIONS EXAM
QUESTIONS AND CORRECT ANSWERS
(VERIFIED ANSWERS) PLUS RATIONALES
2026 Q&A | INSTANT DOWNLOAD PDF.
Core Domains
1. Markov Decision Processes and the Markov Property
2. Policies, Value Functions, and the Bellman Equations
3. Dynamic Programming: Value Iteration and Policy Iteration
4. Model-Free Prediction: Monte Carlo and Temporal-Difference
Learning
5. Model-Free Control: SARSA, Q-Learning, and Exploration
Strategies
6. Function Approximation and Deep Reinforcement Learning
7. Policy Gradient Methods and Actor-Critic Architectures
8. Multi-Armed Bandits and Regret Minimization
9. Game Theory: Nash Equilibria and Mixed Strategies
10. Reward Design, Discounting, and Convergence Properties
Introduction
This comprehensive examination evaluates advanced knowledge of
reinforcement learning principles, algorithms, and their practical
applications. It covers Markov decision processes, dynamic
programming, model-free methods, function approximation, policy
gradients, multi-armed bandits, and game theory. Questions are
,designed as multiple-choice and scenario-based items that require
analysis, computation, and strategic decision-making. The
assessment emphasizes foundational theory, algorithmic
convergence, real-world problem formulation, and the ability to
select appropriate methods for complex sequential decision-making
tasks. Candidates must demonstrate readiness to design, implement,
and evaluate reinforcement learning solutions in academic and
professional settings.
SECTION ONE: QUESTIONS 1–100
1. Which of the following best describes the Markov property
in a Markov Decision Process?
A. The next state depends on the entire history of states and actions
B. The next state depends only on the current state and action
C. The reward depends only on the current state
D. The policy must be deterministic
B. The next state depends only on the current state and action
RATIONALE: The Markov property states that the future is
independent of the past given the present. In an MDP, the transition
probability and reward depend only on the current state and action,
not on the full history.
2. In an MDP, what does the discount factor γ (gamma)
control?
,A. The learning rate
B. The trade-off between immediate and future rewards
C. The exploration rate
D. The number of states
B. The trade-off between immediate and future rewards
RATIONALE: The discount factor γ ∈ [0,1] determines how
much the agent values future rewards. A γ close to 0 makes the
agent short-sighted, while a γ close to 1 makes it far-sighted.
3. What is the Bellman optimality equation for the state-value
function V*(s)?
A. V(s) = max_a Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
B. V(s) = Σ_a π(a|s) Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
C. V(s) = R(s) + γ max_a V(s')
D. V(s) = max_a Q(s,a) + γV(s')
A. V(s) = max_a Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
RATIONALE: The Bellman optimality equation expresses the
value of a state as the maximum over actions of the expected
immediate reward plus the discounted value of the next state.
4. Which algorithm is guaranteed to converge to the optimal
policy for a finite MDP with known dynamics?
A. Q-learning
B. SARSA
, C. Value iteration
D. Monte Carlo control
C. Value iteration
RATIONALE: Value iteration is a dynamic programming
method that converges to the optimal value function and policy
when the MDP model (transition probabilities and rewards) is fully
known.
5. What is the primary difference between value iteration and
policy iteration?
A. Value iteration uses a fixed policy, while policy iteration improves
the policy
B. Value iteration combines policy evaluation and improvement in
one update, while policy iteration alternates between full evaluation
and improvement
C. Policy iteration is model-free, while value iteration is model-
based
D. Value iteration converges in fewer iterations than policy iteration
B. Value iteration combines policy evaluation and improvement
in one update, while policy iteration alternates between full
evaluation and improvement
RATIONALE: Value iteration performs a single sweep of
Bellman optimality updates, effectively combining truncated policy
evaluation with policy improvement. Policy iteration fully evaluates
a policy before improving it.
PROBLEMS AND SOLUTIONS EXAM
QUESTIONS AND CORRECT ANSWERS
(VERIFIED ANSWERS) PLUS RATIONALES
2026 Q&A | INSTANT DOWNLOAD PDF.
Core Domains
1. Markov Decision Processes and the Markov Property
2. Policies, Value Functions, and the Bellman Equations
3. Dynamic Programming: Value Iteration and Policy Iteration
4. Model-Free Prediction: Monte Carlo and Temporal-Difference
Learning
5. Model-Free Control: SARSA, Q-Learning, and Exploration
Strategies
6. Function Approximation and Deep Reinforcement Learning
7. Policy Gradient Methods and Actor-Critic Architectures
8. Multi-Armed Bandits and Regret Minimization
9. Game Theory: Nash Equilibria and Mixed Strategies
10. Reward Design, Discounting, and Convergence Properties
Introduction
This comprehensive examination evaluates advanced knowledge of
reinforcement learning principles, algorithms, and their practical
applications. It covers Markov decision processes, dynamic
programming, model-free methods, function approximation, policy
gradients, multi-armed bandits, and game theory. Questions are
,designed as multiple-choice and scenario-based items that require
analysis, computation, and strategic decision-making. The
assessment emphasizes foundational theory, algorithmic
convergence, real-world problem formulation, and the ability to
select appropriate methods for complex sequential decision-making
tasks. Candidates must demonstrate readiness to design, implement,
and evaluate reinforcement learning solutions in academic and
professional settings.
SECTION ONE: QUESTIONS 1–100
1. Which of the following best describes the Markov property
in a Markov Decision Process?
A. The next state depends on the entire history of states and actions
B. The next state depends only on the current state and action
C. The reward depends only on the current state
D. The policy must be deterministic
B. The next state depends only on the current state and action
RATIONALE: The Markov property states that the future is
independent of the past given the present. In an MDP, the transition
probability and reward depend only on the current state and action,
not on the full history.
2. In an MDP, what does the discount factor γ (gamma)
control?
,A. The learning rate
B. The trade-off between immediate and future rewards
C. The exploration rate
D. The number of states
B. The trade-off between immediate and future rewards
RATIONALE: The discount factor γ ∈ [0,1] determines how
much the agent values future rewards. A γ close to 0 makes the
agent short-sighted, while a γ close to 1 makes it far-sighted.
3. What is the Bellman optimality equation for the state-value
function V*(s)?
A. V(s) = max_a Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
B. V(s) = Σ_a π(a|s) Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
C. V(s) = R(s) + γ max_a V(s')
D. V(s) = max_a Q(s,a) + γV(s')
A. V(s) = max_a Σ_s' P(s'|s,a)[R(s,a,s') + γV(s')]
RATIONALE: The Bellman optimality equation expresses the
value of a state as the maximum over actions of the expected
immediate reward plus the discounted value of the next state.
4. Which algorithm is guaranteed to converge to the optimal
policy for a finite MDP with known dynamics?
A. Q-learning
B. SARSA
, C. Value iteration
D. Monte Carlo control
C. Value iteration
RATIONALE: Value iteration is a dynamic programming
method that converges to the optimal value function and policy
when the MDP model (transition probabilities and rewards) is fully
known.
5. What is the primary difference between value iteration and
policy iteration?
A. Value iteration uses a fixed policy, while policy iteration improves
the policy
B. Value iteration combines policy evaluation and improvement in
one update, while policy iteration alternates between full evaluation
and improvement
C. Policy iteration is model-free, while value iteration is model-
based
D. Value iteration converges in fewer iterations than policy iteration
B. Value iteration combines policy evaluation and improvement
in one update, while policy iteration alternates between full
evaluation and improvement
RATIONALE: Value iteration performs a single sweep of
Bellman optimality updates, effectively combining truncated policy
evaluation with policy improvement. Policy iteration fully evaluates
a policy before improving it.
