Cs7643 Quiz 5 - P1 Questions And Verified Correct Answers Graded A-Mlatest Guaranteed Pass.docx

CS7643 Quiz 5 - P1
EXAM questions and verified
correct answers 100% guaranteed
pass latest 2026-2027



Exploration is more of a long-term benefit concept where it allows the agent to improve its
knowledge about each action which could lead to long term benefit.- This leads to high variance
in gradients and inefficient learning



Exploitation basically exploits the agent's current estimated value and chooses the greedy
approach to get the most reward. However, the agent is being greedy with the estimated value
and not the actual value, so chances are it might not get the most reward.

Since we are always updating the pi gather policy with the greed pi trained policy, it will not
have incentive to explore other less rewarding states that may eventually lead to an overall
higher reward.



Challenge 2: Non iid, highly correlated data

- We are gathering data for this new policy, leading to data bias by the policy. The policy will be
highly correlated with similar states, actions, and reward.



--> store (s, a, s', r) pairs and continually update episodes (older samples discarded)

--> Train Q-Network on random mini batches of transitions from the replay memory instead of
consecutive examples

--> larger the buffer, lower the correlation

,Experience Replay - CORRECT ANSWER--Experience Replay Addresses this:

Queue of past experience from which a minibatch is randomly sampled, buffer if a finite size
and older samples are discarded in favor of new ones.



Increasing size of buffer, lowers correlation between data points sampled in a minibatch



- store (s, a, s', r) pairs and continually update episodes (older samples discarded)



- Train Q-Network on random mini batches of transitions from the replay memory instead of
consecutive examples



Value Based RL Methods Learn, Model Based Methods Learn - CORRECT ANSWER-- Q-Functions

- Transition and reward function



Parameterized Policy - CORRECT ANSWER-theta* = argmax_theta( E [ sum( R(s, a) ) ] )



REINFORCE Algorithm - CORRECT ANSWER-1. Gather data using current policy

- Sample trajectories t by acting according to PI



2. Compute the gradient update

- sum(delta_theta * pi_theta(a_t | s_t)) * sum(R(s_t, a_t))



3. Update Policy Parameters

- theta = theta + alpha*(policy gradient)

, Drawbacks of Policy Gradients - CORRECT ANSWER-- Increases or decreases gradient based on
entire sequence of actions

--> Can't determine which action was good or bad

--> Credit assignent problem

--> This suffers from high variance and unstable learning



- How to reduce variance?

--> subtracting an action independent baseline from the reward preserves the mean of the
gradient expectation while possibly reducing the variance



- Different Choices of baseline result in different variants of Policy Gradient Algorithm



Actor-Critic - CORRECT ANSWER-- Replaces rewards with Q_(PI_theta)(s, a)

- E[delta_theta * log_pi_theta(a | s) (Q_(PI_theta)(s, a))



- Replaces rewards with the q function of the policy that is learned from the data



Advantage Actor-critic - CORRECT ANSWER-- Uses Q minus V values (i.e. Advantage)

- E[delta_theta * log_pi_theta(a | s) (Q_(PI_theta)(s, a) - V_PI_theta)(s))



- Substitutes the reward with the advantage of the policy



Semi-Supervised Learning - CORRECT ANSWER-- Some amounts of labeled data (10/20 per
category), but lots of unlabeled data



Self-Supervised Learning - CORRECT ANSWER-- A form of unsupervised learning

- No labels for representation learning