Page 1 of 82
CS 7641 REINFORCEMENT LEARNING UNIT EXAM PREP 2026
QUESTIONS WITH VERIFIED ANSWERS DETAILED RATIONALES GRADED
A+
Part I: Markov Decision Processes (Questions 1–30)
1. In a Markov Decision Process, which statement about the Markov property is
correct?
A. The next state depends on the entire history of states and actions.
B. The next state depends only on the current state and action, not on the
history.
C. The reward function must be deterministic.
D. The state space must be finite.
Rationale: The Markov property states that the future is independent of the past
given the present; the next state and reward depend only on the current state and
action.
2. Which component of an MDP defines the probability of transitioning from one
state to another given an action?
A. Reward function
B. Policy
C. Transition function
D. Value function
1|Page
,Page 2 of 82
Rationale: The transition function P(s′|s,a) specifies the probability of moving to
state s′ when action a is taken in state s.
3. What is the primary objective of reinforcement learning?
A. To minimize the number of actions taken
B. To maximize the cumulative discounted reward over time
C. To classify states into categories
D. To predict the next state given the current state
Rationale: The RL agent’s goal is to learn a policy that maximizes the expected
sum of discounted rewards over the long run.
4. In a discounted MDP, what is the role of the discount factor γ?
A. It determines the learning rate
B. It controls the trade-off between immediate and future rewards
C. It defines the probability of exploration
D. It measures the accuracy of the value function
Rationale: γ ∈ [0,1) determines how much the agent values future rewards
relative to immediate rewards; a lower γ makes the agent more short-sighted.
5. Which of the following best describes a deterministic policy?
A. A policy that assigns non-zero probability to multiple actions in the same state
B. A policy that maps each state to a single action with probability 1
C. A policy that changes randomly at each step
D. A policy that depends on the entire history of states
Rationale: By definition, a deterministic policy selects the same action for a given
state every time.
6. A stochastic policy differs from a deterministic policy because a stochastic policy
does which of the following?
A. Always selects the same action for a given state
B. Assigns a probability distribution over actions in each state
2|Page
,Page 3 of 82
C. Ignores the current state
D. Maximizes immediate reward only
Rationale: Stochastic policies assign non-zero probability to multiple actions,
allowing randomness in action selection.
7. Which of the following statements about optimal policies is true?
A. The optimal policy is always unique for any MDP.
B. Some MDPs have multiple optimal policies that yield the same expected
return.
C. An optimal policy must be deterministic.
D. There is never an optimal policy for large MDPs.
Rationale: Multiple policies can achieve the same maximal expected return, so
the optimal policy is not necessarily unique.
8. Policy evaluation involves which of the following?
A. Finding the optimal policy directly
B. Computing the state-value or action-value functions for a given policy
C. Selecting the best action in each state without a policy
D. Eliminating all suboptimal actions permanently
Rationale: Policy evaluation computes the value functions for a specific policy,
quantifying its expected return from each state.
9. The policy improvement theorem guarantees which of the following?
A. That a better policy can always be found after each iteration
B. That a policy is at least as good as its predecessor when improved greedily
C. That the optimal policy is unique
D. That random policies are optimal
Rationale: The theorem ensures that greedy improvement yields a policy at least
as good as the previous one; if already optimal, the improvement step returns the
same policy.
3|Page
, Page 4 of 82
10. A random policy that chooses all actions uniformly is generally which of the
following?
A. Optimal for large state spaces
B. Not optimal because it does not maximize expected return
C. Always better than a greedy policy
D. Equivalent to the optimal policy
Rationale: Random policies do not maximize expected return and are generally
suboptimal, though they can aid exploration.
11. In an MDP, the reward function R(s,a,s′) defines which of the following?
A. The probability of reaching state s′
B. The expected immediate reward for transitioning from s to s′ via action a
C. The long-term value of state s
D. The policy’s action distribution
Rationale: The reward function specifies the expected immediate reward
associated with a transition, separate from the transition probability.
12. Which of the following best describes the return G_t in reinforcement
learning?
A. The immediate reward at time t only
B. The sum of future discounted rewards starting from time t
C. The average reward over all episodes
D. The probability of reaching a terminal state
Rationale: Return G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + … includes discounted
future rewards, not just immediate reward.
13. The state-value function V^π(s) represents which of the following?
A. The immediate reward from state s
B. The expected return starting from state s and following policy π
C. The probability of action a in state s
D. The transition probability P(s′|s,a)
4|Page
CS 7641 REINFORCEMENT LEARNING UNIT EXAM PREP 2026
QUESTIONS WITH VERIFIED ANSWERS DETAILED RATIONALES GRADED
A+
Part I: Markov Decision Processes (Questions 1–30)
1. In a Markov Decision Process, which statement about the Markov property is
correct?
A. The next state depends on the entire history of states and actions.
B. The next state depends only on the current state and action, not on the
history.
C. The reward function must be deterministic.
D. The state space must be finite.
Rationale: The Markov property states that the future is independent of the past
given the present; the next state and reward depend only on the current state and
action.
2. Which component of an MDP defines the probability of transitioning from one
state to another given an action?
A. Reward function
B. Policy
C. Transition function
D. Value function
1|Page
,Page 2 of 82
Rationale: The transition function P(s′|s,a) specifies the probability of moving to
state s′ when action a is taken in state s.
3. What is the primary objective of reinforcement learning?
A. To minimize the number of actions taken
B. To maximize the cumulative discounted reward over time
C. To classify states into categories
D. To predict the next state given the current state
Rationale: The RL agent’s goal is to learn a policy that maximizes the expected
sum of discounted rewards over the long run.
4. In a discounted MDP, what is the role of the discount factor γ?
A. It determines the learning rate
B. It controls the trade-off between immediate and future rewards
C. It defines the probability of exploration
D. It measures the accuracy of the value function
Rationale: γ ∈ [0,1) determines how much the agent values future rewards
relative to immediate rewards; a lower γ makes the agent more short-sighted.
5. Which of the following best describes a deterministic policy?
A. A policy that assigns non-zero probability to multiple actions in the same state
B. A policy that maps each state to a single action with probability 1
C. A policy that changes randomly at each step
D. A policy that depends on the entire history of states
Rationale: By definition, a deterministic policy selects the same action for a given
state every time.
6. A stochastic policy differs from a deterministic policy because a stochastic policy
does which of the following?
A. Always selects the same action for a given state
B. Assigns a probability distribution over actions in each state
2|Page
,Page 3 of 82
C. Ignores the current state
D. Maximizes immediate reward only
Rationale: Stochastic policies assign non-zero probability to multiple actions,
allowing randomness in action selection.
7. Which of the following statements about optimal policies is true?
A. The optimal policy is always unique for any MDP.
B. Some MDPs have multiple optimal policies that yield the same expected
return.
C. An optimal policy must be deterministic.
D. There is never an optimal policy for large MDPs.
Rationale: Multiple policies can achieve the same maximal expected return, so
the optimal policy is not necessarily unique.
8. Policy evaluation involves which of the following?
A. Finding the optimal policy directly
B. Computing the state-value or action-value functions for a given policy
C. Selecting the best action in each state without a policy
D. Eliminating all suboptimal actions permanently
Rationale: Policy evaluation computes the value functions for a specific policy,
quantifying its expected return from each state.
9. The policy improvement theorem guarantees which of the following?
A. That a better policy can always be found after each iteration
B. That a policy is at least as good as its predecessor when improved greedily
C. That the optimal policy is unique
D. That random policies are optimal
Rationale: The theorem ensures that greedy improvement yields a policy at least
as good as the previous one; if already optimal, the improvement step returns the
same policy.
3|Page
, Page 4 of 82
10. A random policy that chooses all actions uniformly is generally which of the
following?
A. Optimal for large state spaces
B. Not optimal because it does not maximize expected return
C. Always better than a greedy policy
D. Equivalent to the optimal policy
Rationale: Random policies do not maximize expected return and are generally
suboptimal, though they can aid exploration.
11. In an MDP, the reward function R(s,a,s′) defines which of the following?
A. The probability of reaching state s′
B. The expected immediate reward for transitioning from s to s′ via action a
C. The long-term value of state s
D. The policy’s action distribution
Rationale: The reward function specifies the expected immediate reward
associated with a transition, separate from the transition probability.
12. Which of the following best describes the return G_t in reinforcement
learning?
A. The immediate reward at time t only
B. The sum of future discounted rewards starting from time t
C. The average reward over all episodes
D. The probability of reaching a terminal state
Rationale: Return G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + … includes discounted
future rewards, not just immediate reward.
13. The state-value function V^π(s) represents which of the following?
A. The immediate reward from state s
B. The expected return starting from state s and following policy π
C. The probability of action a in state s
D. The transition probability P(s′|s,a)
4|Page
