HIGH-YIELD REVISION GUIDE & CHEAT SHEET
Reinforcement Learning & LLM Align-
ment Complete Semester Guide
High-Yield Comprehensive Guide, Bellman Equations, Value Iteration Traces,
DPO Hand-Calculations, PPO & Quick-Cram Appendix
Artificial Intelligence & Post-Training Engineering | Course Code: University Of California -
Berkeley - CS 285 | 2024-2025 Comprehensive Edition
, Artificial Intelligence & Post-Training Engineering · Uni-Reinforcement Learning & LLM Alignment Complete Semes-
versity Of California - Berkeley - CS 285 ter Guide
📘 WHAT’S INSIDE THIS GUIDE:
✔ Markov Decision Processes: Formal 5-tuple foundations, discounted returns, and Markov state-
transition matrices
✔ Bellman Optimality Suite: Contraction mapping proofs, Value Iteration vs. Policy Iteration
convergence
✔ WORKED STEP-BY-STEP PROBLEM: Step-by-step GridWorld Value Iteration state update
matrix calculation
✔ Policy Gradient Methods: The Policy Gradient Theorem proof, REINFORCE algorithm, and
Actor-Critic baseline variance reduction
✔ Proximal Policy Optimization (PPO): Clipped surrogate loss bounds [1-eps, 1+eps] and Gener-
alized Advantage Estimation (GAE)
✔ LLM Alignment & Post-Training: Classical 3-stage RLHF (SFT, Bradley-Terry Reward Modeling,
PPO) vs. Direct Preference Optimization (DPO)
✔ WORKED STEP-BY-STEP PROBLEM: Step-by-step hand calculation of DPO cross-entropy
loss and implicit reward margins
✔ 10 Deadly RL Exam Traps (Off-Policy Deadly Triad, Reward Hacking, exploration collapse)
✔ APPENDIX: 1-Hour Emergency Pre-Exam Quick-Cram Cheat Sheet with master Bellman
equations and 25 active recall flashcards
Disclaimer: This is an independently created academic study aid. It is not affiliated with, endorsed by, or sponsored by any specific
university, textbook author, or publisher.
Independent Study Aid. Not affiliated with or endorsed by any specific university or Page 2 of 7
publisher.
Reinforcement Learning & LLM Align-
ment Complete Semester Guide
High-Yield Comprehensive Guide, Bellman Equations, Value Iteration Traces,
DPO Hand-Calculations, PPO & Quick-Cram Appendix
Artificial Intelligence & Post-Training Engineering | Course Code: University Of California -
Berkeley - CS 285 | 2024-2025 Comprehensive Edition
, Artificial Intelligence & Post-Training Engineering · Uni-Reinforcement Learning & LLM Alignment Complete Semes-
versity Of California - Berkeley - CS 285 ter Guide
📘 WHAT’S INSIDE THIS GUIDE:
✔ Markov Decision Processes: Formal 5-tuple foundations, discounted returns, and Markov state-
transition matrices
✔ Bellman Optimality Suite: Contraction mapping proofs, Value Iteration vs. Policy Iteration
convergence
✔ WORKED STEP-BY-STEP PROBLEM: Step-by-step GridWorld Value Iteration state update
matrix calculation
✔ Policy Gradient Methods: The Policy Gradient Theorem proof, REINFORCE algorithm, and
Actor-Critic baseline variance reduction
✔ Proximal Policy Optimization (PPO): Clipped surrogate loss bounds [1-eps, 1+eps] and Gener-
alized Advantage Estimation (GAE)
✔ LLM Alignment & Post-Training: Classical 3-stage RLHF (SFT, Bradley-Terry Reward Modeling,
PPO) vs. Direct Preference Optimization (DPO)
✔ WORKED STEP-BY-STEP PROBLEM: Step-by-step hand calculation of DPO cross-entropy
loss and implicit reward margins
✔ 10 Deadly RL Exam Traps (Off-Policy Deadly Triad, Reward Hacking, exploration collapse)
✔ APPENDIX: 1-Hour Emergency Pre-Exam Quick-Cram Cheat Sheet with master Bellman
equations and 25 active recall flashcards
Disclaimer: This is an independently created academic study aid. It is not affiliated with, endorsed by, or sponsored by any specific
university, textbook author, or publisher.
Independent Study Aid. Not affiliated with or endorsed by any specific university or Page 2 of 7
publisher.
