Key Algorithms: Q-Learning, Policy Gradient, and PPO (Intuition Only)
~13 min read
Three foundational RL algorithm families, explained through intuition rather than equations: learning action VALUES (Q-learning), learning a policy DIRECTLY (Policy Gradient), and learning stably (PPO).
Key Algorithms: Q-Learning, Policy Gradient, and PPO (Intuition Only) is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Q-learning learns a VALUE for every (state, action) pair, then picks actions indirectly by always choosing the highest-value one — struggles with continuous action spaces
- •Policy Gradient methods learn the policy DIRECTLY as action probabilities, naturally fitting continuous or huge discrete action spaces like next-token selection
- •GRPO (covered in sft-vs-rft) belongs to the Policy Gradient family — this is why RFT can handle the enormous 'choose the next token' action space
- •PPO addresses Policy Gradient's instability by limiting ('clipping') how much the policy is allowed to change per update, trading some speed for much greater stability
- •PPO was the dominant algorithm behind RLHF for years; GRPO was later designed as a simpler, cheaper alternative specifically for LLM fine-tuning