Practical Tradeoffs: Data, Compute, Stability and Real Examples

The decision tree from the previous subtopic tells you WHICH method fits your data/task shape; this subtopic covers what you should expect practically once you've made that choice, since SFT and RFT differ substantially in what they demand from you as an engineer.

Data requirements differ in KIND, not just amount. SFT needs a dataset where every example already has its correct completion written out — this is often the expensive part: either paying human annotators to write high-quality completions, or building a synthetic-data pipeline (like this course's own Distilabel example earlier in this fine-tuning chapter, where two LLMs generate candidate responses and a third judges and picks the best one). RFT instead needs a RELIABLE reward function — often easier to build for verifiable tasks (checking a final math answer is much simpler than writing a full worked solution by hand), but genuinely hard or impossible to build well for subjective tasks, which is exactly why the decision tree routes non-verifiable tasks to RLHF instead of RFT.

Compute cost differs because of HOW each method uses the model during training. SFT computes a loss directly from one forward pass per example (predict, compare to label, backprop) — comparatively cheap and highly parallelizable across a fixed dataset. RFT requires the model to GENERATE candidate outputs during training (often multiple per prompt, to give GRPO something to compare within a group — hence 'Group' Relative Policy Optimization) before it can even compute a reward and update — generation is far more expensive than a single forward pass, so RFT training runs typically cost meaningfully more compute for a comparable amount of wall-clock training data seen.

Stability also differs. SFT's fixed targets make training behavior predictable and easy to debug — loss curves are smooth and comparable across runs, since every example's 'correct answer' never changes. RFT's online, exploratory nature makes it more prone to instabilities that don't exist in SFT — reward hacking (the model finding a way to score well on the reward function without actually solving the task correctly, exploiting a loophole in how the reward is defined) is the classic failure mode, along with training that can be noisier and more sensitive to hyperparameters, since the model is chasing a moving target of its own recent behavior rather than a fixed label.

This course's own worked example (from the GRPO walkthrough immediately following this section) ties these tradeoffs together concretely: fine-tuning a model for math reasoning with GRPO uses four deterministic reward functions (match format exactly, match format approximately, check the answer, check numbers) specifically BECAUSE these are cheap and hard to game — a practical illustration that RFT's extra complexity is worth paying specifically when you can design a reward function this reliable; when you can't, the decision tree's routing back to SFT or RLHF reflects a real practical tradeoff, not just a theoretical one.

# A side-by-side cost/stability comparison, illustrating WHY RFT costs
# more compute per training example than SFT.

def sft_cost_per_example(forward_pass_cost: float = 1.0) -> float:
    """SFT: one forward pass, compare to fixed label, backprop."""
    return forward_pass_cost  # a single generation, directly scored against the label

def rft_cost_per_example(forward_pass_cost: float = 1.0, group_size: int = 8) -> float:
    """RFT/GRPO: must GENERATE multiple candidate completions per prompt
    (a 'group') before a relative reward comparison is even possible."""
    return forward_pass_cost * group_size   # generation is the expensive part

print(f"SFT cost per example: {sft_cost_per_example():.1f}x")
print(f"RFT cost per example (group_size=8): {rft_cost_per_example():.1f}x")

def check_for_reward_hacking(model_output: str, expected_pattern: str) -> bool:
    """A simplified stand-in for the book's 'match format exactly' /
    'match format approximately' reward checks -- designed to be hard
    to game, which is exactly why RFT needs RELIABLE reward functions."""
    exact_match = model_output.strip() == expected_pattern
    # A model reward-hacking a weak checker might just echo the pattern
    # without solving anything -- a well-designed reward function guards
    # against this by also checking the underlying computation, not just format
    return exact_match

print(check_for_reward_hacking("<answer>4</answer>", "<answer>4</answer>"))