GRPO: Group Relative Policy Optimization
Master DeepSeek's reinforcement learning algorithm, group relative advantages, and training loops.
Fine-Tuning Controls:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Explain how GRPO replaces the Critic model with group statistics
- •Compute relative advantages from a group of rewards
- •Design verification-based reward functions
What is it?
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm, introduced by DeepSeek, that fine-tunes LLMs for reasoning-heavy tasks like math, logic, and code — without needing a separately-trained reward model or human-labeled completions. It generates a GROUP of candidate outputs per prompt, scores each with a deterministic reward function (does the answer match, does the code pass its tests), and updates the policy based on each candidate's reward RELATIVE to the group's average. This makes it dramatically simpler and cheaper than classic PPO-based RLHF, and it's the core technique behind reasoning models like DeepSeek-R1.
Why it exists
PPO RLHF requires running a separate Critic model to estimate state values, which takes up massive GPU memory. GRPO removes it.
Problem it solves
VRAM bottlenecks in reinforcement learning, training instability, and complex reward calibration.
Intuition
Instead of hiring a separate teacher to score your work on a absolute scale, you write 4 drafts, check how well they pass tests, and grade them relative to your own average.
Analogy
GRPO is like grading on a curve: you do not need an external reference standard, you just reward students who did better than the class average and penalize those who did worse.
Technical explanation
GRPO samples $G$ outputs ${o_1, o_2, ..., o_g}$ for a prompt. Advantage is $A_i = rac{r_i - mean(R)}{std(R)}$. Gradients optimize: $L_{GRPO}( heta) = rac{1}{G} sum minleft(r_i( heta) A_i, ext{clip}(r_i( heta), 1-epsilon, 1+epsilon) A_i ight) - eta D_{KL}(pi_ heta || pi_{ref})$.
Architecture
Actor policy model node interacting with reward calculators and advantage normalization nodes without Critic parameters.
Workflow
- Generate group samples -> 2. Score via rewards -> 3. Normalize to advantages -> 4. Run policy updates.
Example
rewards = [0, 1, 1, 0]; mean = 0.5; std = 0.577 advantages = [ (x-0.5)/0.577 for x in rewards ]
Advantages: [-0.86, 0.86, 0.86, -0.86]
Real-world usage
Training DeepSeek-R1-Zero to output chain-of-thought inside <think> tags by rewarding format adherence.
Trade-offs
Saves VRAM but increases active generation token requirements during training iterations.
Visual explanation
GRPO Advantage flow: Actor samples -> [Output 1, Output 2, Output 3, Output 4] ──(Rewards: 1, 0, 1, 0)──> Compute Mean/Std ──> Normalized Advantages
Advantages
- —
Saves 30-40% GPU VRAM during RL training
- —
Extremely stable for logical math/coding reasoning tasks
Disadvantages
- —
Requires large group sizes (G=4 to 8) to get stable standard deviations
Common mistakes
- —
Using standard deviation without guarding against std=0 (causes division by zero error)
- —
Not adding a reference model KL penalty, causing output collapse
🎤 Interview questions
Explain how GRPO calculates the loss function. How is the policy gradient updated w.r.t the reference model?
📂 Subtopics
Why GRPO: The SFT vs. RFT vs. RLHF Decision Tree
GRPO exists to solve reasoning-heavy tasks where correctness is verifiable — the book's own decision tree shows exactly when GRPO-style RFT is the right call versus SFT or RLHF.
~15 min
How GRPO Works: Groups, Rewards, and the GRPO Loss
GRPO generates multiple candidate responses per prompt (a 'group'), scores each with reward functions, and uses those aggregated rewards to compute gradients that improve the model's reasoning over time.
~15 min
Reward Signal Design: 4 Deterministic Reward Functions
The book's actual GRPO implementation uses 4 specific reward functions — matching format exactly, matching format approximately, checking the answer, and checking numbers — no manual labeling required.
~15 min
Hands-On: Fine-Tuning a Reasoning Model with Unsloth and HuggingFace TRL
The book's actual GRPO walkthrough: load Qwen3-4B-Base with Unsloth, configure LoRA, prepare the Open R1 Math dataset, define the 4 reward functions, and train with HuggingFace TRL's GRPOConfig and GRPOTrainer.
~20 min