CS 7643 DEEP LEARNING QUIZ 4
COMPLETE REVIEW WITH FULL
SOLUTIONS
●● RL: Evaluative Feedback
Answer: - Pick an action, receive a reward
- No supervision for what the correct action is or would have been
(unlike supervised learning)
●● RL: Sequential Decisions
Answer: - Plan and execution actions over a sequence of states
- Reward may be delayed, requiring optimization of future rewards
(long-term planning)
●● Signature Challenges in RL
Answer: Evaluative Feedback: Need trial and error to find the right
action. Requiring agent to try all actions at states.
Delayed Feedback: Actions may not lead to immediate reward
Non-stationarity: Data distribution of visited states changes when the
policy changes.
,Fleeting Nature: of online data (may only see data once). Making it
difficult to learn from past mistakes.
●● MDP
Answer: Framework underlying RL
S: Set of states
A: Set of actions
R: Distribution of Rewards give state S action a
T: Transition probability
y: Discount property
Markov Property: Current state completely characterizes state of the
environment
* Distribution of possible next states give state s and action a does not
depend on previous states or actions.
*Rewards of MDPs are given more weight earlier in the time steps vs at
later time steps.
●● Deterministic vs. Stochastic
Answer: - the next state of the environment is completely determined by
the current state and the action executed by the agent (N Dim Tensor)
- a probability distribution of actions given set of n states and finite set
of m actions. (N x M Dim Tensor)
, *When determining optimal policy. Gamma plays a large role Large
gamma prefers larger rewards where smaller gamma prefers rewards that
are near/close.
●● RL: Equations relating optimal quantities
Answer: 1. V*(S) = max_a(Q*(s, a)
2. PI*(s) = argmax_a(Q*(s, a)
●● V*(S) - Value Function
Answer: max_a (sum_(s') { p(s'|s, a) [r(s, a) + yV*(s')] } )
- Tells us how good is this state
- Likelihood of win/lose from particular state
- Optimal value at a state is the same as the max Q value over all
possible actions at that state
●● Q*(s,a)
Answer: sum_(s') { p(s'|s, a) [r(s, a) + y*max_(a'){Q*(s', a') ] }
- How good is the state-action pair
- In this state, what is the impact of this action on my future?
COMPLETE REVIEW WITH FULL
SOLUTIONS
●● RL: Evaluative Feedback
Answer: - Pick an action, receive a reward
- No supervision for what the correct action is or would have been
(unlike supervised learning)
●● RL: Sequential Decisions
Answer: - Plan and execution actions over a sequence of states
- Reward may be delayed, requiring optimization of future rewards
(long-term planning)
●● Signature Challenges in RL
Answer: Evaluative Feedback: Need trial and error to find the right
action. Requiring agent to try all actions at states.
Delayed Feedback: Actions may not lead to immediate reward
Non-stationarity: Data distribution of visited states changes when the
policy changes.
,Fleeting Nature: of online data (may only see data once). Making it
difficult to learn from past mistakes.
●● MDP
Answer: Framework underlying RL
S: Set of states
A: Set of actions
R: Distribution of Rewards give state S action a
T: Transition probability
y: Discount property
Markov Property: Current state completely characterizes state of the
environment
* Distribution of possible next states give state s and action a does not
depend on previous states or actions.
*Rewards of MDPs are given more weight earlier in the time steps vs at
later time steps.
●● Deterministic vs. Stochastic
Answer: - the next state of the environment is completely determined by
the current state and the action executed by the agent (N Dim Tensor)
- a probability distribution of actions given set of n states and finite set
of m actions. (N x M Dim Tensor)
, *When determining optimal policy. Gamma plays a large role Large
gamma prefers larger rewards where smaller gamma prefers rewards that
are near/close.
●● RL: Equations relating optimal quantities
Answer: 1. V*(S) = max_a(Q*(s, a)
2. PI*(s) = argmax_a(Q*(s, a)
●● V*(S) - Value Function
Answer: max_a (sum_(s') { p(s'|s, a) [r(s, a) + yV*(s')] } )
- Tells us how good is this state
- Likelihood of win/lose from particular state
- Optimal value at a state is the same as the max Q value over all
possible actions at that state
●● Q*(s,a)
Answer: sum_(s') { p(s'|s, a) [r(s, a) + y*max_(a'){Q*(s', a') ] }
- How good is the state-action pair
- In this state, what is the impact of this action on my future?
