How GRPO Works: Groups, Rewards, and the GRPO Loss

Group Relative Policy Optimization is a reinforcement learning method that fine-tunes LLMs for math and reasoning tasks using deterministic reward functions, eliminating the need for labeled data. This course gives a brief but precise overview of the actual mechanics.

The process starts with a dataset and a reasoning-focused system prompt — something like 'Think step by step…' — which nudges the model toward showing its reasoning rather than jumping straight to an answer. From there: the LLM generates MULTIPLE candidate responses to the same prompt using a sampling engine — this multiplicity is exactly the 'group' in Group Relative Policy Optimization's name. Rather than generating one response and scoring it in isolation, GRPO generates a whole group of candidates for the same input.

Each response in that group is then assigned rewards — from deterministic reward functions, covered in depth in the next subtopic — and these rewards are aggregated to produce a score for every generated response. This is the 'relative' part of the name: rather than needing an absolute, externally-calibrated notion of 'how good is this response,' GRPO can score each candidate RELATIVE to the other candidates in its own group, which is what lets it work without a separately-trained reward model the way PPO (the algorithm behind RLHF) typically needs.

Finally, a GRPO loss function uses these rewards to calculate gradients, backpropagation updates the LLM's weights, and the model improves its reasoning ability over time — candidates that scored better relative to their group get reinforced, candidates that scored worse get discouraged, repeated across many prompts and training steps. HuggingFace's TRL library provides everything this GRPO diagram describes out of the box, via GRPOConfig and GRPOTrainer, which is exactly what the hands-on subtopic later in this topic walks through using directly.

# Conceptual sketch of one GRPO training step — generate a GROUP of
# candidates, score them, and update based on RELATIVE performance
# within the group (real training uses HuggingFace TRL's GRPOTrainer).

def grpo_training_step(policy_model, prompt: str, reward_fns: list, group_size: int = 8):
    # 1) Generate a GROUP of candidate responses for the same prompt
    candidates = [policy_model.generate(prompt) for _ in range(group_size)]

    # 2) Score each candidate — rewards from multiple deterministic
    #    reward functions, aggregated per candidate (next subtopic)
    rewards = [
        sum(reward_fn(candidate, prompt) for reward_fn in reward_fns)
        for candidate in candidates
    ]

    # 3) "Group relative": normalize each reward against the GROUP's
    #    own mean/std, not an externally-calibrated absolute score —
    #    this is what avoids needing a separately-trained reward model
    import statistics
    mean_r, std_r = statistics.mean(rewards), statistics.pstdev(rewards) or 1.0
    advantages = [(r - mean_r) / std_r for r in rewards]

    # 4) The GRPO loss uses these advantages to compute gradients,
    #    reinforcing above-average candidates, discouraging below-average ones
    policy_model.grpo_update(prompt, candidates, advantages)