RL for Language Models: How RLHF Uses RL, and the Connection to GRPO
~12 min read
Mapping RL's abstract vocabulary onto LLM training concretely: the model IS the policy, generating text IS taking actions, and human/reward-function feedback IS the reward signal RLHF and GRPO both optimize against.
RL for Language Models: How RLHF Uses RL, and the Connection to GRPO is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •In LLM training, the model IS the RL agent/policy: its weights map current text (state) to next-token probabilities (action distribution)
- •Generating a whole response is a full sequence of RL actions, one token at a time — making LLM generation structurally an RL problem, not just a metaphor
- •RLHF trains a reward MODEL on human preference comparisons (needed when a task isn't automatically verifiable), then uses that model's score as the RL reward signal
- •GRPO instead uses a deterministic reward FUNCTION directly (for automatically verifiable tasks like math/code), skipping the learned reward model entirely
- •GRPO's 'Group Relative' design generates multiple candidate outputs per prompt and rewards them relative to the group's average, avoiding the separate value function classic PPO needs