Training Stages: Pretraining, SFT, RLHF, GRPO
Deconstruct the four stages of building a production reasoning LLM from scratch.
Fine-Tuning Controls:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Contrast Pre-training and Supervised Fine-Tuning (SFT)
- •Describe RLHF using Reward Models and PPO optimization
- •Explain DeepSeek's Group Relative Policy Optimization (GRPO) and how it eliminates the critic model
What is it?
LLM training is the multi-stage pipeline that turns a raw next-token predictor into a useful, safe conversational assistant: pretraining teaches broad language ability and world knowledge from massive text corpora; supervised fine-tuning (SFT) teaches the model to follow instructions using labeled prompt-completion pairs; and RLHF or GRPO-style reinforcement fine-tuning aligns the model's behavior to human preferences or verifiable task rewards. Each stage solves a different problem, and understanding where one stage's job ends and the next begins is essential to knowing which technique to reach for when customizing a model.
Why it exists
Base models output random web text. Alignment is necessary to make them follow user instructions and answer accurately and safely.
Problem it solves
Uncontrolled text generation, dangerous/malicious responses, lack of dialogue skills, and logical reasoning gaps.
Intuition
Building an LLM is like raising a student. First, in "Pretraining", the student reads the entire library to learn words and general knowledge. In "SFT", the student is taught specific questions and answers by a tutor. In "RLHF", the student writes answers, and humans rate them to reward good behavior. In "GRPO", the model generates a group of answers, and is rewarded based on how accurate or logical they are.
Analogy
PPO is hiring a personal critic to grade every step you take. GRPO is looking at 5 variations of your own work, ranking them, and learning from your group average without a third-party critic.
Technical explanation
Stage 1 — Pretraining: next-token prediction on trillions of tokens. Loss = −(1/T) Σ log P(t_i | t_<i). Trained with AdamW + cosine LR schedule + gradient clipping (norm ≤ 1.0). Produces a base model that completes text but does not follow instructions.
Stage 2 — Supervised Fine-Tuning (SFT): train on (instruction, response) pairs with teacher-forcing — same cross-entropy loss but only computed on assistant tokens. Typically 1–3 epochs on 10K–1M pairs suffice; more causes forgetting.
Stage 3 — Reward Modeling: a separate model R(prompt, response) → scalar. Trained on human preference pairs (chosen ≻ rejected) using Bradley-Terry loss: L = −log σ(R(chosen) − R(rejected)).
Stage 4a — PPO (RLHF): the SFT model is the actor; reward model scores completions; a KL penalty term (β·KL[π_actor ‖ π_SFT]) prevents the actor from drifting too far. Stage 4b — GRPO (DeepSeek): eliminates the value/critic network. Samples G responses per prompt, normalizes rewards within the group: Â_i = (r_i − μ_G) / σ_G, then updates policy with clipped surrogate loss. Uses 30–40% less VRAM than PPO.
Architecture
Full training stack (Chip Huyen Ch.2): Pretraining cluster: 1000s of GPUs, FSDP/Megatron-LM model parallelism, NVLink for fast all-reduce, checkpointing every 1K steps. SFT stack: single-node or DDP, LoRA adapters reduce VRAM 8×, Axolotl/LLaMA-Factory frameworks. RLHF stack (PPO): Actor model + Reference model + Reward model + Critic/Value model — 4 copies of the base model in memory simultaneously. GRPO stack: Actor model + Reference model + Reward function — no critic, dramatically simpler. Used by DeepSeek-R1, Qwen, and Kimi K1.5. Reward functions can be rule-based (format check, compiler pass/fail) instead of a neural RM — this is called RLAIF or process reward modeling.
Workflow
- Load base pre-trained model -> 2. Run SFT instructions -> 3. Sample output groups -> 4. Calculate relative advantages -> 5. Run policy gradient updates.
Example
GRPO reward normalization (core math)
import torch
def grpo_advantages(rewards: torch.Tensor) -> torch.Tensor: # rewards: (G,) — one scalar per sampled completion mu, sigma = rewards.mean(), rewards.std().clamp(min=1e-8) return (rewards - mu) / sigma # normalized advantage
SFT loss — only compute on assistant tokens
def sft_loss(logits, labels, ignore_index=-100): # labels[i] = -100 for system/user tokens, real token_id for assistant tokens return F.cross_entropy(logits.view(-1, logits.size(-1)), labels.view(-1), ignore_index=ignore_index)
KL penalty term in PPO
kl = (log_probs_actor - log_probs_ref).sum(-1) # per-token KL reward_with_kl = reward - beta * kl # β typically 0.05–0.2
Real-world usage
Llama 3's training pipeline (Meta AI): ~15T tokens pretraining on 16K H100s, followed by iterative SFT + DPO (Direct Preference Optimization — a reward-free RLHF variant). DeepSeek-R1: starts from a base model, applies GRPO directly on math/code verification rewards (compiler pass = +1, wrong = −1) — no human preference data needed. Mistral/Mixtral: SFT only (no RLHF), relying on high-quality instruction datasets. In practice: 80% of quality comes from SFT data quality. RLHF adds the last 20% (reduces refusals, improves format, enhances safety). Bad SFT data cannot be fixed by RLHF.
Trade-offs
GRPO trade-off: reduces training VRAM by 30-40% but requires generating multiple samples per prompt (computational bottleneck during training).
Visual explanation
PPO vs GRPO Architecture Comparison: PPO: [Actor] ──(Generates)──> [Tokens] <──(Evaluates)── [Critic] (Requires heavy VRAM) [Reward Model] and [Reference Model] also active in GPU memory
GRPO: [Actor] ──(Generates Group: A, B, C, D)──> [Relative Advantage Calculation] No Critic model needed! Standard deviation scales rewards directly.
Advantages
- —
Extremely low memory footprint for reinforcement training
- —
Stable learning curves for logical tasks
Disadvantages
- —
Requires structured reward logic (like compilers or format matchers)
- —
Vulnerable to reward hacking
🎤 Interview questions
Compare standard PPO RLHF with GRPO. What are the key mathematical differences and hardware savings?
📂 Subtopics
Stage 1 — Pre-training: Teaching the Basics of Language
Before any fine-tuning, an LLM starts as a randomly initialized model that knows nothing. Pre-training teaches it grammar, world facts, and next-token prediction by training on massive text corpora — but leaves it unable to hold a conversation.
~15 min
Stage 2 — Instruction Fine-Tuning: Teaching the Model to Follow Prompts
Instruction fine-tuning (IFT) turns a text-completing pre-trained model into a conversational assistant by training it on instruction-response pairs, teaching it to answer questions, follow formatting, and summarize rather than just continue text.
~15 min
Stage 3 — Preference Fine-Tuning & RLHF: Aligning with Human Preferences
The 'which response do you prefer?' screen you've seen on ChatGPT is how preference data gets collected. A reward model learns to predict human preference, and the LLM is then updated with reinforcement learning (RLHF) — usually via the PPO algorithm.
~20 min
Stage 4 — Reasoning Fine-Tuning & GRPO: Reinforcement Learning with Verifiable Rewards
For tasks with a single correct answer (math, logic), you don't need human preference data — correctness itself is the reward signal. This is Reinforcement Learning with Verifiable Rewards (RLVR), and GRPO (used by DeepSeek) is a popular technique for it.
~20 min