Stage 4 — Reasoning Fine-Tuning & GRPO: Reinforcement Learning with Verifiable Rewards

RLHF (the previous stage) is built around human preference — useful when quality is subjective. But for reasoning tasks like math and logic problems, there's usually exactly one correct final answer and a well-defined series of steps to reach it. For these tasks, you don't need a human to compare two responses and say which is better — you can just check correctness directly and use that as the reward signal.

This is reasoning fine-tuning, and the general technique is called Reinforcement Learning with Verifiable Rewards (RLVR). The loop is straightforward: the model generates an answer to a prompt, that answer is compared against the known correct answer, and a reward is assigned based purely on whether it's correct (and often partial credit for correct reasoning steps even if the final answer is slightly off). No reward model or human annotator is needed in the loop at all — correctness is objectively checkable.

GRPO (Group Relative Policy Optimization), developed by DeepSeek, is a popular technique for this stage. Rather than needing a separately trained reward model like PPO does, GRPO generates a group of multiple candidate answers for the same prompt, scores each one against the verifiable ground truth, and updates the model to favor whichever answers in the group scored best relative to the others in that same group — hence 'group relative.' This sidesteps the cost and complexity of training and maintaining a separate reward model.

Put together, the 4 stages form the complete pipeline: start from a randomly initialized model, pre-train it on large-scale corpora to learn language itself, use instruction fine-tuning to make it follow commands, then use preference fine-tuning (RLHF) and reasoning fine-tuning (RLVR/GRPO) to sharpen its responses along the two axes that matter most: subjective human alignment, and objective correctness on verifiable reasoning tasks.

# Simplified RLVR reward function + GRPO-style group scoring —
# no reward model needed, since correctness is directly checkable.

def verify_math_answer(model_answer: str, ground_truth: str) -> float:
    # Correctness IS the reward — no learned reward model required
    return 1.0 if model_answer.strip() == ground_truth.strip() else 0.0

def grpo_step(policy_model, prompt: str, ground_truth: str, group_size: int = 8):
    # Generate a GROUP of candidate answers for the same prompt
    candidates = [policy_model.generate(prompt) for _ in range(group_size)]
    rewards = [verify_math_answer(c, ground_truth) for c in candidates]

    # "Group relative": normalize each reward against the group's own
    # mean/std, rather than needing a separately trained reward model
    mean_r = sum(rewards) / len(rewards)
    advantages = [r - mean_r for r in rewards]

    # Update the policy to favor candidates that scored above the
    # group average, and away from those that scored below it
    policy_model.grpo_update(prompt, candidates, advantages)