Stage 3 — Preference Fine-Tuning & RLHF: Aligning with Human Preferences
~20 min read
The 'which response do you prefer?' screen you've seen on ChatGPT is how preference data gets collected. A reward model learns to predict human preference, and the LLM is then updated with reinforcement learning (RLHF) — usually via the PPO algorithm.
Stage 3 — Preference Fine-Tuning & RLHF: Aligning with Human Preferences is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •PFT collects human preference data via A/B comparisons — exactly the 'which response do you prefer?' UI pattern seen in ChatGPT
- •A reward model is trained to predict human preference from these comparisons
- •The LLM is then updated via reinforcement learning, guided by the reward model's scores — this combination is called RLHF
- •PPO (Proximal Policy Optimization) is the algorithm most commonly used to actually apply the reward signal to the LLM's weights
- •RLHF aligns tone, helpfulness, and safety — subjective qualities with no single 'correct' answer — but doesn't specifically optimize objectively-verifiable tasks like math