CS 7641 RL Unit Quiz Problems and Solutions ||
Update | 100% Correct - GT - PERFECT RATIONALE
Q1: CS 7641 RL - What is Markov Decision Process MDP definition?
A. No discount factor
B. Only states and actions
C. Only rewards
D. MDP defined as tuple (S,A,P,R,gamma): S states, A actions, P transition probability P(s'|s,a), R reward R(s,a,s'), gamma discount
factor 0<=gamma<=1
Answer & ULTRA-Detailed Rationale:
D. MDP defined as tuple (S,A,P,R,gamma): S states, A actions, P transition probability P(s'|s,a), R reward R(s,a,s'), gamma discount
factor 0<=gamma<=1
CONCEPT: MDP definition CS 7641. CORRECT: MDP Markov Decision Process defined as 5-tuple (S,A,P,R,gamma): S finite set
states, A finite set actions, P transition function P(s'|s,a) probability next state s' given state s action a, R reward function R(s,a,s')
immediate reward, gamma discount factor 0<=gamma<=1 - Markov property future depends only current state. WHY B WRONG: Need
S A P R gamma not only states actions. WHY C WRONG: Gamma discount factor required 0-1. WHY D WRONG: Need all components
S A P R gamma. TIP: MDP tuple (S,A,P,R,gamma) S states A actions P transition P(s'|s,a) R reward R(s,a,s') gamma discount Markov
property.
Q2: CS 7641 RL - What is Bellman equation for state value V pi?
A. No expectation over pi
B. V(s)=R only
C. No gamma
D. Bellman expectation equation: V^pi(s) = sum_a pi(a|s) sum_s' P(s'|s,a)[R(s,a,s') + gamma V^pi(s')]
Answer & ULTRA-Detailed Rationale:
D. Bellman expectation equation: V^pi(s) = sum_a pi(a|s) sum_s' P(s'|s,a)[R(s,a,s') + gamma V^pi(s')]
CONCEPT: Bellman equation V pi. CORRECT: Bellman expectation equation for state value function V^pi(s) under policy pi: V^pi(s)=
sum_a pi(a|s) sum_{s'} P(s'|s,a)[R(s,a,s') + gamma V^pi(s')] - recursive expected return. WHY B WRONG: Not only R includes future
discounted V. WHY C WRONG: Expectation over policy pi(a|s). WHY D WRONG: Gamma discount future. TIP: Bellman V^pi(s)= sum_a
pi(a|s) sum_s' P(s'|s,a)[R+gamma V^pi(s')] recursive expected return.
Q3: CS 7641 RL - What is Bellman optimality equation for V star?
A. No max
B. Bellman optimality: V*(s)= max_a sum_s' P(s'|s,a)[R(s,a,s') + gamma V*(s')]
C. V*(s)= min_a
D. V*(s)= sum_a average
Answer & ULTRA-Detailed Rationale:
B. Bellman optimality: V*(s)= max_a sum_s' P(s'|s,a)[R(s,a,s') + gamma V*(s')]
CONCEPT: Bellman optimality V*. CORRECT: Bellman optimality equation for optimal value V*: V*(s)= max_a sum_{s'}
P(s'|s,a)[R(s,a,s') + gamma V*(s')] - optimal policy chooses action maximizing expected return. WHY B WRONG: Max not average sum.
WHY C WRONG: Max over actions required optimality. WHY D WRONG: Max not min maximizing reward. TIP: Bellman optimality
V*(s)= max_a sum_s' P(s'|s,a)[R+gamma V*(s')] max action optimal.
Page 1 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
,Q4: CS 7641 RL - What is Q-learning update rule?
A. Q-learning off-policy TD update: Q(s,a) <- Q(s,a) + alpha[ r + gamma max_{a'} Q(s',a') - Q(s,a) ] - alpha learning rate
B. No max_{a'}
C. On-policy SARSA update
D. Q(s,a)=r only
Answer & ULTRA-Detailed Rationale:
A. Q-learning off-policy TD update: Q(s,a) <- Q(s,a) + alpha[ r + gamma max_{a'} Q(s',a') - Q(s,a) ] - alpha learning rate
CONCEPT: Q-learning update. CORRECT: Q-learning off-policy temporal difference update: Q(s,a) <- Q(s,a) + alpha[ r + gamma
max_{a'} Q(s',a') - Q(s,a) ] where alpha learning rate 0-1, r immediate reward, gamma discount, max_{a'} Q(s',a') bootstrap optimal future
- off-policy because max not necessarily policy action. WHY B WRONG: Not only r includes future max Q. WHY C WRONG: Max_{a'}
required off-policy. WHY D WRONG: SARSA on-policy Q(s',a') not max. TIP: Q-learning off-policy Q<-Q+alpha[r+gamma max_a'
Q(s',a')-Q] alpha learning rate max future bootstrap.
Q5: CS 7641 RL - What is difference between Q-learning off-policy and SARSA on-policy?
A. Q-learning off-policy updates with max_{a'} Q(s',a') regardless of behavior policy, SARSA on-policy updates with Q(s',a') where a'
is actual next action taken by policy - SARSA: Q<-Q+alpha[r+gamma Q(s',a')-Q]
B. Same update
C. Q-learning on-policy SARSA off-policy
D. No difference
Answer & ULTRA-Detailed Rationale:
A. Q-learning off-policy updates with max_{a'} Q(s',a') regardless of behavior policy, SARSA on-policy updates with Q(s',a') where a' is
actual next action taken by policy - SARSA: Q<-Q+alpha[r+gamma Q(s',a')-Q]
CONCEPT: Q-learning vs SARSA. CORRECT: Q-learning off-policy: Q(s,a)<-Q+alpha[r+gamma max_{a'} Q(s',a')-Q] learns optimal
policy regardless of behavior policy, SARSA on-policy: Q(s,a)<-Q+alpha[r+gamma Q(s',a')-Q] where a' is actual next action taken
following current policy - on-policy learns policy being followed - acronym State Action Reward State Action. WHY B WRONG: Different
updates max vs actual next action. WHY C WRONG: Q-learning off-policy SARSA on-policy not opposite. WHY D WRONG: Difference
exists off vs on policy. TIP: Q-learning off-policy max_a' Q(s',a') learns optimal regardless behavior, SARSA on-policy Q(s',a') actual next
action State Action Reward State Action learns policy followed.
Q6: CS 7641 RL - What is exploration vs exploitation and epsilon-greedy?
A. Exploration vs exploitation tradeoff: exploitation choose best known action max Q, exploration try random action to discover better
- epsilon-greedy: with prob epsilon explore random, with prob 1-epsilon exploit greedy max Q - epsilon decays
B. Only exploration
C. No epsilon-greedy
D. Only exploitation
Answer & ULTRA-Detailed Rationale:
A. Exploration vs exploitation tradeoff: exploitation choose best known action max Q, exploration try random action to discover better -
epsilon-greedy: with prob epsilon explore random, with prob 1-epsilon exploit greedy max Q - epsilon decays
CONCEPT: Exploration exploitation epsilon-greedy. CORRECT: Exploration vs exploitation tradeoff: exploitation choose best known
action max Q to maximize reward, exploration try random action to discover better policy - epsilon-greedy strategy: with probability
epsilon choose random action explore, with probability 1-epsilon choose greedy action max Q exploit - epsilon often decays over time
1.0 to 0.1. WHY B WRONG: Need both exploration and exploitation. WHY C WRONG: Need both not only exploration. WHY D WRONG:
Epsilon-greedy exists exploration vs exploitation. TIP: Exploration vs exploitation tradeoff exploitation max Q best known exploration
random discover better epsilon-greedy prob epsilon random 1-epsilon greedy max Q epsilon decays.
Page 2 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
,Q7: CS 7641 RL - What is discount factor gamma effect?
A. Gamma always 0
B. No effect
C. Gamma >1 allowed infinite
D. Gamma discount factor 0<=gamma<=1: gamma near 0 myopic short-sighted immediate reward, gamma near 1 far-sighted
long-term reward - gamma=0 only immediate r, gamma=1 no discount infinite horizon may diverge
Answer & ULTRA-Detailed Rationale:
D. Gamma discount factor 0<=gamma<=1: gamma near 0 myopic short-sighted immediate reward, gamma near 1 far-sighted long-term
reward - gamma=0 only immediate r, gamma=1 no discount infinite horizon may diverge
CONCEPT: Discount factor gamma. CORRECT: Discount factor gamma 0<=gamma<=1 effect: gamma near 0 myopic agent values
immediate reward only, gamma near 1 far-sighted values long-term future rewards - gamma=0 V=R only immediate, gamma=1 no
discount future undiscounted may diverge infinite horizon requires gamma<1 for convergence - GT CS7641. WHY B WRONG: Effect
myopic vs far-sighted. WHY C WRONG: Gamma must be 0-1 >1 diverges >1 not allowed infinite horizon. WHY D WRONG: Gamma
varies 0-1 not always 0. TIP: Gamma 0<=gamma<=1 gamma near 0 myopic immediate gamma near 1 far-sighted long-term gamma 0
only immediate gamma 1 no discount infinite horizon may diverge.
Q8: CS 7641 RL - What is policy iteration vs value iteration?
A. Same algorithm
B. Value iteration no max
C. Policy iteration: 1) policy evaluation compute V^pi, 2) policy improvement greedy pi'(s)=argmax_a Q^pi(s,a) repeat until stable -
Value iteration: combines evaluation improvement V<- max_a sum P[R+gamma V] iterative - both find optimal
D. Policy iteration no evaluation
Answer & ULTRA-Detailed Rationale:
C. Policy iteration: 1) policy evaluation compute V^pi, 2) policy improvement greedy pi'(s)=argmax_a Q^pi(s,a) repeat until stable - Value
iteration: combines evaluation improvement V<- max_a sum P[R+gamma V] iterative - both find optimal
CONCEPT: Policy iteration vs value iteration. CORRECT: Policy iteration: iterative two steps 1) policy evaluation compute V^pi(s) for
current pi via Bellman, 2) policy improvement make greedy pi'(s)=argmax_a sum_s' P[R+gamma V^pi(s')] - repeat until policy stable
optimal. Value iteration: directly iterates Bellman optimality V_{k+1}(s)= max_a sum_s' P(s'|s,a)[R+gamma V_k(s')] - combines
evaluation improvement one step - both converge to V* pi*. WHY B WRONG: Different policy iteration two steps evaluation improvement
value iteration one step max. WHY C WRONG: Policy iteration has evaluation compute V^pi. WHY D WRONG: Value iteration has max.
TIP: Policy iteration evaluation V^pi then improvement greedy argmax Q^pi repeat stable optimal, Value iteration V_{k+1}= max_a sum
P[R+gamma V_k] combines.
Page 3 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
, Q9: CS 7641 RL - What is Monte Carlo vs Temporal Difference TD learning?
A. TD no bootstrap
B. Monte Carlo bootstraps
C. Same
D. Monte Carlo learns from complete episodes return G_t = sum gamma^k R_{t+k} updates after episode, TD learns from incomplete
bootstraps using V(s') - e.g., TD(0) V<-V+alpha[r+gamma V(s')-V] - MC no bias high variance, TD low variance bias
Answer & ULTRA-Detailed Rationale:
D. Monte Carlo learns from complete episodes return G_t = sum gamma^k R_{t+k} updates after episode, TD learns from incomplete
bootstraps using V(s') - e.g., TD(0) V<-V+alpha[r+gamma V(s')-V] - MC no bias high variance, TD low variance bias
CONCEPT: Monte Carlo vs TD. CORRECT: Monte Carlo RL learns from complete episodes actual return G_t = sum_{k} gamma^k
R_{t+k+1} updates V(s)<-V+alpha[G_t - V(s)] after episode ends - no bootstrapping - unbiased high variance. Temporal Difference TD
learns from incomplete episodes bootstraps using estimate V(s') - TD(0): V(s)<-V(s)+alpha[r+gamma V(s')-V(s)] - low variance some
bias - GT CS7641. WHY B WRONG: Different MC complete episodes no bootstrap vs TD incomplete bootstrap. WHY C WRONG:
Monte Carlo no bootstrap uses actual return. WHY D WRONG: TD bootstraps uses V(s'). TIP: Monte Carlo complete episodes G_t sum
gamma^k R updates after episode no bootstrap unbiased high variance, TD incomplete bootstraps V(s') TD(0) V<-V+alpha[r+gamma
V(s')-V] low variance bias.
Q10: CS 7641 RL - What is function approximation in RL and DQN?
A. Function approximation uses neural network to approximate Q(s,a) for large continuous state spaces - DQN Deep Q-Network uses
CNN approximates Q(s,a;theta) with experience replay and target network to stabilize - loss = (r+gamma max_a'
Q(s',a';theta-)-Q(s,a;theta))^2
B. Only tabular Q
C. No experience replay
D. No neural network
Answer & ULTRA-Detailed Rationale:
A. Function approximation uses neural network to approximate Q(s,a) for large continuous state spaces - DQN Deep Q-Network uses
CNN approximates Q(s,a;theta) with experience replay and target network to stabilize - loss = (r+gamma max_a'
Q(s',a';theta-)-Q(s,a;theta))^2
CONCEPT: Function approximation DQN. CORRECT: Function approximation in RL for large continuous state spaces tabular infeasible
uses parameterized function e.g., neural network to approximate Q(s,a;theta) - DQN Deep Q-Network Mnih 2015 uses deep CNN
approximates Q(s,a) with two tricks: experience replay buffer random samples breaks correlation, target network theta- fixed updates
periodically stabilizes - loss MSE between target r+gamma max_a' Q(s',a';theta-) and Q(s,a;theta). WHY B WRONG: Tabular only small
discrete function approximation for large continuous. WHY C WRONG: Neural network used DQN. WHY D WRONG: Experience replay
used breaks correlation stabilizes. TIP: Function approximation neural network Q(s,a;theta) large continuous DQN CNN Q(s,a;theta)
experience replay random samples target network theta- fixed periodic loss MSE target r+gamma max_a' Q(s',a';theta-) - Q(s,a;theta).
Page 4 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
Update | 100% Correct - GT - PERFECT RATIONALE
Q1: CS 7641 RL - What is Markov Decision Process MDP definition?
A. No discount factor
B. Only states and actions
C. Only rewards
D. MDP defined as tuple (S,A,P,R,gamma): S states, A actions, P transition probability P(s'|s,a), R reward R(s,a,s'), gamma discount
factor 0<=gamma<=1
Answer & ULTRA-Detailed Rationale:
D. MDP defined as tuple (S,A,P,R,gamma): S states, A actions, P transition probability P(s'|s,a), R reward R(s,a,s'), gamma discount
factor 0<=gamma<=1
CONCEPT: MDP definition CS 7641. CORRECT: MDP Markov Decision Process defined as 5-tuple (S,A,P,R,gamma): S finite set
states, A finite set actions, P transition function P(s'|s,a) probability next state s' given state s action a, R reward function R(s,a,s')
immediate reward, gamma discount factor 0<=gamma<=1 - Markov property future depends only current state. WHY B WRONG: Need
S A P R gamma not only states actions. WHY C WRONG: Gamma discount factor required 0-1. WHY D WRONG: Need all components
S A P R gamma. TIP: MDP tuple (S,A,P,R,gamma) S states A actions P transition P(s'|s,a) R reward R(s,a,s') gamma discount Markov
property.
Q2: CS 7641 RL - What is Bellman equation for state value V pi?
A. No expectation over pi
B. V(s)=R only
C. No gamma
D. Bellman expectation equation: V^pi(s) = sum_a pi(a|s) sum_s' P(s'|s,a)[R(s,a,s') + gamma V^pi(s')]
Answer & ULTRA-Detailed Rationale:
D. Bellman expectation equation: V^pi(s) = sum_a pi(a|s) sum_s' P(s'|s,a)[R(s,a,s') + gamma V^pi(s')]
CONCEPT: Bellman equation V pi. CORRECT: Bellman expectation equation for state value function V^pi(s) under policy pi: V^pi(s)=
sum_a pi(a|s) sum_{s'} P(s'|s,a)[R(s,a,s') + gamma V^pi(s')] - recursive expected return. WHY B WRONG: Not only R includes future
discounted V. WHY C WRONG: Expectation over policy pi(a|s). WHY D WRONG: Gamma discount future. TIP: Bellman V^pi(s)= sum_a
pi(a|s) sum_s' P(s'|s,a)[R+gamma V^pi(s')] recursive expected return.
Q3: CS 7641 RL - What is Bellman optimality equation for V star?
A. No max
B. Bellman optimality: V*(s)= max_a sum_s' P(s'|s,a)[R(s,a,s') + gamma V*(s')]
C. V*(s)= min_a
D. V*(s)= sum_a average
Answer & ULTRA-Detailed Rationale:
B. Bellman optimality: V*(s)= max_a sum_s' P(s'|s,a)[R(s,a,s') + gamma V*(s')]
CONCEPT: Bellman optimality V*. CORRECT: Bellman optimality equation for optimal value V*: V*(s)= max_a sum_{s'}
P(s'|s,a)[R(s,a,s') + gamma V*(s')] - optimal policy chooses action maximizing expected return. WHY B WRONG: Max not average sum.
WHY C WRONG: Max over actions required optimality. WHY D WRONG: Max not min maximizing reward. TIP: Bellman optimality
V*(s)= max_a sum_s' P(s'|s,a)[R+gamma V*(s')] max action optimal.
Page 1 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
,Q4: CS 7641 RL - What is Q-learning update rule?
A. Q-learning off-policy TD update: Q(s,a) <- Q(s,a) + alpha[ r + gamma max_{a'} Q(s',a') - Q(s,a) ] - alpha learning rate
B. No max_{a'}
C. On-policy SARSA update
D. Q(s,a)=r only
Answer & ULTRA-Detailed Rationale:
A. Q-learning off-policy TD update: Q(s,a) <- Q(s,a) + alpha[ r + gamma max_{a'} Q(s',a') - Q(s,a) ] - alpha learning rate
CONCEPT: Q-learning update. CORRECT: Q-learning off-policy temporal difference update: Q(s,a) <- Q(s,a) + alpha[ r + gamma
max_{a'} Q(s',a') - Q(s,a) ] where alpha learning rate 0-1, r immediate reward, gamma discount, max_{a'} Q(s',a') bootstrap optimal future
- off-policy because max not necessarily policy action. WHY B WRONG: Not only r includes future max Q. WHY C WRONG: Max_{a'}
required off-policy. WHY D WRONG: SARSA on-policy Q(s',a') not max. TIP: Q-learning off-policy Q<-Q+alpha[r+gamma max_a'
Q(s',a')-Q] alpha learning rate max future bootstrap.
Q5: CS 7641 RL - What is difference between Q-learning off-policy and SARSA on-policy?
A. Q-learning off-policy updates with max_{a'} Q(s',a') regardless of behavior policy, SARSA on-policy updates with Q(s',a') where a'
is actual next action taken by policy - SARSA: Q<-Q+alpha[r+gamma Q(s',a')-Q]
B. Same update
C. Q-learning on-policy SARSA off-policy
D. No difference
Answer & ULTRA-Detailed Rationale:
A. Q-learning off-policy updates with max_{a'} Q(s',a') regardless of behavior policy, SARSA on-policy updates with Q(s',a') where a' is
actual next action taken by policy - SARSA: Q<-Q+alpha[r+gamma Q(s',a')-Q]
CONCEPT: Q-learning vs SARSA. CORRECT: Q-learning off-policy: Q(s,a)<-Q+alpha[r+gamma max_{a'} Q(s',a')-Q] learns optimal
policy regardless of behavior policy, SARSA on-policy: Q(s,a)<-Q+alpha[r+gamma Q(s',a')-Q] where a' is actual next action taken
following current policy - on-policy learns policy being followed - acronym State Action Reward State Action. WHY B WRONG: Different
updates max vs actual next action. WHY C WRONG: Q-learning off-policy SARSA on-policy not opposite. WHY D WRONG: Difference
exists off vs on policy. TIP: Q-learning off-policy max_a' Q(s',a') learns optimal regardless behavior, SARSA on-policy Q(s',a') actual next
action State Action Reward State Action learns policy followed.
Q6: CS 7641 RL - What is exploration vs exploitation and epsilon-greedy?
A. Exploration vs exploitation tradeoff: exploitation choose best known action max Q, exploration try random action to discover better
- epsilon-greedy: with prob epsilon explore random, with prob 1-epsilon exploit greedy max Q - epsilon decays
B. Only exploration
C. No epsilon-greedy
D. Only exploitation
Answer & ULTRA-Detailed Rationale:
A. Exploration vs exploitation tradeoff: exploitation choose best known action max Q, exploration try random action to discover better -
epsilon-greedy: with prob epsilon explore random, with prob 1-epsilon exploit greedy max Q - epsilon decays
CONCEPT: Exploration exploitation epsilon-greedy. CORRECT: Exploration vs exploitation tradeoff: exploitation choose best known
action max Q to maximize reward, exploration try random action to discover better policy - epsilon-greedy strategy: with probability
epsilon choose random action explore, with probability 1-epsilon choose greedy action max Q exploit - epsilon often decays over time
1.0 to 0.1. WHY B WRONG: Need both exploration and exploitation. WHY C WRONG: Need both not only exploration. WHY D WRONG:
Epsilon-greedy exists exploration vs exploitation. TIP: Exploration vs exploitation tradeoff exploitation max Q best known exploration
random discover better epsilon-greedy prob epsilon random 1-epsilon greedy max Q epsilon decays.
Page 2 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
,Q7: CS 7641 RL - What is discount factor gamma effect?
A. Gamma always 0
B. No effect
C. Gamma >1 allowed infinite
D. Gamma discount factor 0<=gamma<=1: gamma near 0 myopic short-sighted immediate reward, gamma near 1 far-sighted
long-term reward - gamma=0 only immediate r, gamma=1 no discount infinite horizon may diverge
Answer & ULTRA-Detailed Rationale:
D. Gamma discount factor 0<=gamma<=1: gamma near 0 myopic short-sighted immediate reward, gamma near 1 far-sighted long-term
reward - gamma=0 only immediate r, gamma=1 no discount infinite horizon may diverge
CONCEPT: Discount factor gamma. CORRECT: Discount factor gamma 0<=gamma<=1 effect: gamma near 0 myopic agent values
immediate reward only, gamma near 1 far-sighted values long-term future rewards - gamma=0 V=R only immediate, gamma=1 no
discount future undiscounted may diverge infinite horizon requires gamma<1 for convergence - GT CS7641. WHY B WRONG: Effect
myopic vs far-sighted. WHY C WRONG: Gamma must be 0-1 >1 diverges >1 not allowed infinite horizon. WHY D WRONG: Gamma
varies 0-1 not always 0. TIP: Gamma 0<=gamma<=1 gamma near 0 myopic immediate gamma near 1 far-sighted long-term gamma 0
only immediate gamma 1 no discount infinite horizon may diverge.
Q8: CS 7641 RL - What is policy iteration vs value iteration?
A. Same algorithm
B. Value iteration no max
C. Policy iteration: 1) policy evaluation compute V^pi, 2) policy improvement greedy pi'(s)=argmax_a Q^pi(s,a) repeat until stable -
Value iteration: combines evaluation improvement V<- max_a sum P[R+gamma V] iterative - both find optimal
D. Policy iteration no evaluation
Answer & ULTRA-Detailed Rationale:
C. Policy iteration: 1) policy evaluation compute V^pi, 2) policy improvement greedy pi'(s)=argmax_a Q^pi(s,a) repeat until stable - Value
iteration: combines evaluation improvement V<- max_a sum P[R+gamma V] iterative - both find optimal
CONCEPT: Policy iteration vs value iteration. CORRECT: Policy iteration: iterative two steps 1) policy evaluation compute V^pi(s) for
current pi via Bellman, 2) policy improvement make greedy pi'(s)=argmax_a sum_s' P[R+gamma V^pi(s')] - repeat until policy stable
optimal. Value iteration: directly iterates Bellman optimality V_{k+1}(s)= max_a sum_s' P(s'|s,a)[R+gamma V_k(s')] - combines
evaluation improvement one step - both converge to V* pi*. WHY B WRONG: Different policy iteration two steps evaluation improvement
value iteration one step max. WHY C WRONG: Policy iteration has evaluation compute V^pi. WHY D WRONG: Value iteration has max.
TIP: Policy iteration evaluation V^pi then improvement greedy argmax Q^pi repeat stable optimal, Value iteration V_{k+1}= max_a sum
P[R+gamma V_k] combines.
Page 3 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
, Q9: CS 7641 RL - What is Monte Carlo vs Temporal Difference TD learning?
A. TD no bootstrap
B. Monte Carlo bootstraps
C. Same
D. Monte Carlo learns from complete episodes return G_t = sum gamma^k R_{t+k} updates after episode, TD learns from incomplete
bootstraps using V(s') - e.g., TD(0) V<-V+alpha[r+gamma V(s')-V] - MC no bias high variance, TD low variance bias
Answer & ULTRA-Detailed Rationale:
D. Monte Carlo learns from complete episodes return G_t = sum gamma^k R_{t+k} updates after episode, TD learns from incomplete
bootstraps using V(s') - e.g., TD(0) V<-V+alpha[r+gamma V(s')-V] - MC no bias high variance, TD low variance bias
CONCEPT: Monte Carlo vs TD. CORRECT: Monte Carlo RL learns from complete episodes actual return G_t = sum_{k} gamma^k
R_{t+k+1} updates V(s)<-V+alpha[G_t - V(s)] after episode ends - no bootstrapping - unbiased high variance. Temporal Difference TD
learns from incomplete episodes bootstraps using estimate V(s') - TD(0): V(s)<-V(s)+alpha[r+gamma V(s')-V(s)] - low variance some
bias - GT CS7641. WHY B WRONG: Different MC complete episodes no bootstrap vs TD incomplete bootstrap. WHY C WRONG:
Monte Carlo no bootstrap uses actual return. WHY D WRONG: TD bootstraps uses V(s'). TIP: Monte Carlo complete episodes G_t sum
gamma^k R updates after episode no bootstrap unbiased high variance, TD incomplete bootstraps V(s') TD(0) V<-V+alpha[r+gamma
V(s')-V] low variance bias.
Q10: CS 7641 RL - What is function approximation in RL and DQN?
A. Function approximation uses neural network to approximate Q(s,a) for large continuous state spaces - DQN Deep Q-Network uses
CNN approximates Q(s,a;theta) with experience replay and target network to stabilize - loss = (r+gamma max_a'
Q(s',a';theta-)-Q(s,a;theta))^2
B. Only tabular Q
C. No experience replay
D. No neural network
Answer & ULTRA-Detailed Rationale:
A. Function approximation uses neural network to approximate Q(s,a) for large continuous state spaces - DQN Deep Q-Network uses
CNN approximates Q(s,a;theta) with experience replay and target network to stabilize - loss = (r+gamma max_a'
Q(s',a';theta-)-Q(s,a;theta))^2
CONCEPT: Function approximation DQN. CORRECT: Function approximation in RL for large continuous state spaces tabular infeasible
uses parameterized function e.g., neural network to approximate Q(s,a;theta) - DQN Deep Q-Network Mnih 2015 uses deep CNN
approximates Q(s,a) with two tricks: experience replay buffer random samples breaks correlation, target network theta- fixed updates
periodically stabilizes - loss MSE between target r+gamma max_a' Q(s',a';theta-) and Q(s,a;theta). WHY B WRONG: Tabular only small
discrete function approximation for large continuous. WHY C WRONG: Neural network used DQN. WHY D WRONG: Experience replay
used breaks correlation stabilizes. TIP: Function approximation neural network Q(s,a;theta) large continuous DQN CNN Q(s,a;theta)
experience replay random samples target network theta- fixed periodic loss MSE target r+gamma max_a' Q(s',a';theta-) - Q(s,a;theta).
Page 4 | CS 7641 RL UNIT QUIZ - PERFECT 200 UNIQUE - BIG TITLE - GT 100% CORRECT
