Stage 3 — Preference Fine-Tuning & RLHF: Aligning with Human Preferences

You've likely seen a ChatGPT-style interface ask you to choose between two responses to the same prompt. That's not just casual feedback — it's how valuable human preference data gets collected at scale, and it's the raw material for this training stage.

In preference fine-tuning (PFT), a human annotator is shown two candidate responses to the same prompt and picks the one they prefer. Collecting many such comparisons produces a preference dataset. A separate reward model is then trained on this dataset with one job: given a prompt and a response, predict how much a human would like it. Once the reward model exists, the LLM itself is updated using reinforcement learning — the LLM generates a response, the reward model scores it, and the LLM's weights are nudged to make higher-scoring responses more likely in the future.

This whole process is called RLHF — Reinforcement Learning with Human Feedback — and the specific algorithm most commonly used to actually update the model's weights based on the reward signal is PPO (Proximal Policy Optimization). RLHF is what teaches a model to align with human preferences even in situations where there's no single objectively 'correct' answer — tone, helpfulness, safety, and style are all things RLHF shapes, not things a simple instruction-response pair can fully specify.

RLHF is a genuine step up from instruction fine-tuning, but it's not the final stage. It's excellent for subjective quality (tone, safety, helpfulness) but doesn't specifically optimize for tasks that have a single, objectively verifiable correct answer — like math or logic problems. That gap is exactly what the next stage, reasoning fine-tuning, is built to close.

# Simplified reward-model-guided PPO update loop (conceptual) — real
# RLHF pipelines (e.g. trl's PPOTrainer) handle the KL-penalty and
# batching details, but this is the core reward-then-update idea.

def rlhf_step(policy_model, reward_model, prompts: list[str]):
    responses = [policy_model.generate(p) for p in prompts]

    # The reward model, trained on human A/B preference comparisons,
    # scores how much a human would likely prefer each response.
    rewards = [reward_model.score(p, r) for p, r in zip(prompts, responses)]

    # PPO nudges the policy model's weights to make high-reward
    # responses more likely next time, while staying close to its
    # original behavior (a KL penalty prevents the model from
    # collapsing into reward-hacking degenerate outputs).
    policy_model.ppo_update(prompts, responses, rewards)