Why GRPO: The SFT vs. RFT vs. RLHF Decision Tree

GRPO doesn't exist in a vacuum — it's a specific answer to a specific question in a larger decision tree covered elsewhere in this curriculum: given a fine-tuning task, should you use SFT, RFT, or RLHF? Understanding WHY GRPO is the right tool requires understanding what problem the other two don't solve.

This course's decision tree: start by checking whether you have labeled (ground-truth) data. If you don't, the next question is whether the task is verifiable — if not verifiable (correctness can't be automatically checked), you need RLHF, since humans must provide preference signals; if it IS verifiable, RFT works because correctness can be automatically checked, with no human labeling needed at all. If you DO have labeled data, the choice depends on quantity: large datasets favor SFT, while tiny datasets favor RFT specifically when reasoning (like Chain-of-Thought) genuinely helps the task.

GRPO is one of the most effective RFT methods specifically for reasoning-heavy tasks like math or logic — exactly the verifiable-correctness branch of that decision tree. The fundamental problem GRPO (and RFT methods generally) solves: SFT uses static data and often ends up memorizing specific answers rather than learning generalizable reasoning, because it's always trained against fixed, pre-written correct answers. RFT, being online rather than static, has the model actually explore different candidate outputs and learns from a reward signal that scores their correctness — this exploration is what lets the model genuinely learn new reasoning strategies rather than just recalling memorized patterns.

This is exactly why GRPO fine-tuning uses deterministic reward functions to validate a response and assign a reward, eliminating the need for labeled data entirely — for a math problem, you don't need a human to write out one 'correct' worked solution to imitate; you just need a way to automatically CHECK whether a given candidate answer is correct, which is exactly what 'verifiable' means in the decision tree above.

def choose_finetuning_approach(
    has_labeled_data: bool, task_is_verifiable: bool, dataset_size: str,
    reasoning_helps: bool = False,
) -> str:
    """The book's exact decision tree for SFT vs RFT vs RLHF."""
    if not has_labeled_data:
        return "RLHF (humans provide preference signals)" if not task_is_verifiable \
            else "RFT / GRPO (correctness automatically checked, no labels needed)"

    if dataset_size == "large":
        return "SFT"
    # tiny dataset
    return "RFT / GRPO (reasoning helps)" if reasoning_helps else "SFT"

# A math reasoning task with no labeled solutions, but correctness
# is automatically checkable — exactly GRPO's sweet spot
print(choose_finetuning_approach(
    has_labeled_data=False, task_is_verifiable=True, dataset_size="n/a",
))  # -> RFT / GRPO