Reinforcement Fine-Tuning (RFT): Reward Signals Instead of Static Labels

Where the previous subtopic's SFT trains a model to REPRODUCE one specific correct completion, RFT trains a model to DISCOVER correct answers on its own, guided by a reward rather than a label. This course states the RFT process directly: RFT uses an online 'reward' approach — no static labels required. The model explores different outputs, and a Reward Function scores their correctness. Over time, the model learns to generate higher-reward answers using GRPO (Group Relative Policy Optimization, already covered in depth in this curriculum's grpo-reasoning topic).

The word 'online' is the crucial contrast with SFT's 'static.' In SFT, the training data (prompts AND their correct completions) is entirely fixed before training begins. In RFT, the model's OWN outputs, generated live during training, are what get evaluated — the model tries something, a Reward Function grades how good that attempt was, and the model updates itself to make higher-scoring attempts more likely next time. Nobody had to write down the 'correct' completion in advance; they only had to write a function capable of RECOGNIZING a correct (or better) outcome when it sees one — often a much easier thing to specify. For a math problem, you don't need someone to write out one canonical correct solution; you just need a function that checks whether the final numeric answer matches.

This is exactly why this course pairs RFT with GRPO for reasoning-heavy tasks: GRPO is one of the most effective RFT methods for math and logic specifically because those tasks have automatically-checkable outcomes (the reward functions this course uses for its own GRPO walkthrough — matching format exactly, matching format approximately, checking the answer, checking numbers — are all deterministic, needing no human judgment at all). The model can attempt a problem MANY different ways (different reasoning chains, different phrasing) and get rewarded for reaching the right answer via any of them, rather than being penalized for not matching one specific pre-written solution.

This course's summary line captures the core contrast crisply: 'SFT uses static data and often memorizes answers. RFT, being online, learns from rewards and explores new strategies.' RFT trades the simplicity and stability of a fixed target for the ability to genuinely discover new, sometimes better, ways of reasoning through a problem — at the cost of needing a reliable reward function and generally more compute (since the model must generate and evaluate its own attempts repeatedly during training, rather than reading straight through a fixed dataset once).

# Illustrating RFT's core loop: the model EXPLORES different outputs,
# a Reward Function scores them (no static label needed), and the
# model favors higher-reward attempts over time.
import random

def reward_function(question: str, model_answer: str, correct_answer: str) -> float:
    """A deterministic, automatically-checkable reward -- no human-
    written 'correct completion' needed, just a way to VERIFY correctness.
    Mirrors the book's own reward functions: 'check the answer'."""
    return 1.0 if model_answer.strip() == correct_answer.strip() else 0.0

def explore_attempts(question: str, num_attempts: int = 4) -> list[str]:
    """The model EXPLORES different outputs for the same prompt --
    something SFT's teacher-forcing never does."""
    candidate_reasoning_paths = [
        "Let me add: 2+2=4",
        "Counting on fingers: 1,2,3,4 -> answer is 4",
        "2+2 equals 5",             # a wrong attempt -- still explored, just penalized
        "Using multiplication trick: 2*2=4, so 2+2=4 too",
    ]
    return random.sample(candidate_reasoning_paths, min(num_attempts, len(candidate_reasoning_paths)))

question, correct_answer = "What is 2+2?", "4"
for attempt in explore_attempts(question):
    # extract just the final numeric answer from each differently-worded attempt
    extracted_answer = "4" if "4" in attempt and "5" not in attempt else attempt[-1]
    reward = reward_function(question, extracted_answer, correct_answer)
    print(f"attempt={attempt!r:55} reward={reward}")
# Over many training steps, GRPO nudges the model toward whichever
# REASONING STYLES tend to earn reward=1.0, without ever being told
# one single 'correct' phrasing to reproduce