CS7643 LAST QUIZ WITH COMPLETE
SOLUTIONS
Reinforcement llearning l- lcorrect lanswers l-Sequential ldecision lmaking lin lan
lenvironment lwith levaluative lfeedback
Environment: lmay lbe lunknown, lnon-linear, lstochastic land lcomplex
Agent: llearns la lpolicy lto lmap lstates lof lthe lenvironments lto lactions
- lseeks lto lmaximize llong-term lreward
RL: lEvaluative lFeedback l- lcorrect lanswers l-- lPick lan laction, lreceive la lreward
- lNo lsupervision lfor lwhat lthe lcorrect laction lis lor lwould lhave lbeen l(unlike lsupervised
llearning)
, RL: lSequential lDecisions l- lcorrect lanswers l-- lPlan land lexecution lactions lover la
lsequence lof lstates
- lReward lmay lbe ldelayed, lrequiring loptimization lof lfuture lrewards l(long-term
lplanning)
Signature lChallenges lin lRL l- lcorrect lanswers l-Evaluative lFeedback: lNeed ltrial land
lerror lto lfind lthe lright laction
Delayed lFeedback: lActions lmay lnot llead lto limmediate lreward
Non-stationarity: lData ldistribution lof lvisited lstates lchanges lwhen lthe lpolicy lchanges
Fleeting lNature: lof lonline ldata l(may lonly lsee ldata lonce)
MDP l- lcorrect lanswers l-Framework lunderlying lRL
S: lSet lof lstates
A: lSet lof lactions
R: lDistribution lof lRewards
T: lTransition lprobabiliity
y: lDiscount lproperty
Markov lProperty: lCurrent lstate lcompletely lcharacterizes lstate lof lthe lenvironment
RL: lEquations lrelating loptimal lquantities l- lcorrect lanswers l-1. lV*(S) l= lmax_a(Q*(s,
la)
2. lPI*(s) l= largmax_a(Q*(s, la)
V*(S) l- lcorrect lanswers l-max_a l(sum_(s') l{ lp(s'|s, la) l[r(s, la) l+ lyV*(s')] l} l)
Q*(s,a) l- lcorrect lanswers l-sum_(s') l{ lp(s'|s, la) l[r(s, la) l+ ly*max_(a'){Q*(s', la') l] l}
Value lIteration l- lcorrect lanswers l-v_(i+1) l= lmax_a l(sum_(s') l{ lp(s'|s, la) l[r(s, la) l+
lyV_(i)(s')] l} l)
- lrepeat luntil lconvergence
- lTime lcomplexity lper literation lO(|S^2| l|A|)
Policy lIteration l- lcorrect lanswers l-Policy lEvaluation: lCompute lV(pi)
Policy lRefinement: lGreedily lchange laction las lper lV(Pi) lat lnext lstates
Why ldo lPolicy lIteration: lPI_i loften lconverges lto lPI* lsooner lthan lV_PI lto lV_PI*
- lthus lrequires lfew literations
Deep lQ-Learning l- lcorrect lanswers l-- lQ(s, la; lw, lb) l= lw_a^t l* ls l+ lb_a
MSE lLoss l:= l(Q_new(s, la) l- l(r l+ ly*max_a(Q_old(s', la)))^2
SOLUTIONS
Reinforcement llearning l- lcorrect lanswers l-Sequential ldecision lmaking lin lan
lenvironment lwith levaluative lfeedback
Environment: lmay lbe lunknown, lnon-linear, lstochastic land lcomplex
Agent: llearns la lpolicy lto lmap lstates lof lthe lenvironments lto lactions
- lseeks lto lmaximize llong-term lreward
RL: lEvaluative lFeedback l- lcorrect lanswers l-- lPick lan laction, lreceive la lreward
- lNo lsupervision lfor lwhat lthe lcorrect laction lis lor lwould lhave lbeen l(unlike lsupervised
llearning)
, RL: lSequential lDecisions l- lcorrect lanswers l-- lPlan land lexecution lactions lover la
lsequence lof lstates
- lReward lmay lbe ldelayed, lrequiring loptimization lof lfuture lrewards l(long-term
lplanning)
Signature lChallenges lin lRL l- lcorrect lanswers l-Evaluative lFeedback: lNeed ltrial land
lerror lto lfind lthe lright laction
Delayed lFeedback: lActions lmay lnot llead lto limmediate lreward
Non-stationarity: lData ldistribution lof lvisited lstates lchanges lwhen lthe lpolicy lchanges
Fleeting lNature: lof lonline ldata l(may lonly lsee ldata lonce)
MDP l- lcorrect lanswers l-Framework lunderlying lRL
S: lSet lof lstates
A: lSet lof lactions
R: lDistribution lof lRewards
T: lTransition lprobabiliity
y: lDiscount lproperty
Markov lProperty: lCurrent lstate lcompletely lcharacterizes lstate lof lthe lenvironment
RL: lEquations lrelating loptimal lquantities l- lcorrect lanswers l-1. lV*(S) l= lmax_a(Q*(s,
la)
2. lPI*(s) l= largmax_a(Q*(s, la)
V*(S) l- lcorrect lanswers l-max_a l(sum_(s') l{ lp(s'|s, la) l[r(s, la) l+ lyV*(s')] l} l)
Q*(s,a) l- lcorrect lanswers l-sum_(s') l{ lp(s'|s, la) l[r(s, la) l+ ly*max_(a'){Q*(s', la') l] l}
Value lIteration l- lcorrect lanswers l-v_(i+1) l= lmax_a l(sum_(s') l{ lp(s'|s, la) l[r(s, la) l+
lyV_(i)(s')] l} l)
- lrepeat luntil lconvergence
- lTime lcomplexity lper literation lO(|S^2| l|A|)
Policy lIteration l- lcorrect lanswers l-Policy lEvaluation: lCompute lV(pi)
Policy lRefinement: lGreedily lchange laction las lper lV(Pi) lat lnext lstates
Why ldo lPolicy lIteration: lPI_i loften lconverges lto lPI* lsooner lthan lV_PI lto lV_PI*
- lthus lrequires lfew literations
Deep lQ-Learning l- lcorrect lanswers l-- lQ(s, la; lw, lb) l= lw_a^t l* ls l+ lb_a
MSE lLoss l:= l(Q_new(s, la) l- l(r l+ ly*max_a(Q_old(s', la)))^2
