Supervised Fine-Tuning (SFT): Static Labels, Matching Completions

Before comparing SFT and RFT, it helps to be precise about what SFT actually does, since RFT is defined largely in CONTRAST to it. This course lays out the SFT process in three steps: it starts with a static labeled dataset of prompt-completion pairs, adjusts the model weights to match these completions, and the best model (a LoRA checkpoint, since SFT is typically done via LoRA or a similar PEFT method rather than full fine-tuning) is then deployed for inference.

Unpacking each piece: 'static labeled dataset' means someone (a human annotator, or a synthetic-data pipeline like the Distilabel process covered earlier in this course's own fine-tuning chapter) has already decided, in advance, exactly what the CORRECT output should be for each input prompt. This dataset doesn't change during training — it's fixed before training starts, hence 'static.' Training then becomes a straightforward supervised-learning problem: for each prompt, generate a prediction, compare it against the known-correct completion (using cross-entropy loss, from the neural-networks prerequisite unit), and adjust weights to make future predictions closer to that target. This is sometimes called teacher-forcing: during training, the model is always shown the CORRECT previous tokens (from the labeled completion) when predicting the next one, rather than its own possibly-wrong previous predictions — keeping training stable and directly comparable across examples.

The key property that defines SFT, and that this course highlights directly: SFT uses static data and often memorizes answers. The model is never rewarded for finding a DIFFERENT correct path to the answer — it's explicitly trained to reproduce the specific completion in the dataset. This makes SFT excellent when you already know exactly what a good answer looks like for every training example (customer support responses in your company's tone, code following your team's style conventions, structured data extraction with a known correct format) — SFT is the right tool whenever 'correct' can be written down in advance as a specific target completion.

Where SFT struggles is tasks where there are MANY valid paths to a correct answer (math reasoning, multi-step planning) — memorizing one specific worked solution doesn't teach the model to discover ITS OWN correct reasoning path on a new, unseen problem nearly as well as being rewarded for reaching correct answers by any valid route. That distinction is exactly what motivates RFT, covered next.

# Illustrating the core SFT training step: predict, compare against
# the FIXED labeled completion, compute loss, adjust weights.
# (Cross-entropy loss is from the neural-networks prerequisite unit.)
import math

def cross_entropy(true_token_probs: dict, predicted_probs: dict) -> float:
    """Same cross-entropy loss from the neural-networks unit --
    minimized when predicted_probs matches the labeled target."""
    return -sum(
        true_token_probs[t] * math.log(max(predicted_probs.get(t, 1e-12), 1e-12))
        for t in true_token_probs
    )

# A static labeled dataset -- fixed BEFORE training starts, exactly
# as the book describes. Each entry has ONE correct completion.
sft_dataset = [
    {"prompt": "Refund policy?", "completion": "Refunds are processed within 5 business days."},
    {"prompt": "2 + 2 = ?",       "completion": "4"},
]

def sft_training_step(prompt: str, labeled_completion: str, model_predict_fn) -> float:
    """Teacher-forcing: the model predicts the next token, but is always
    SHOWN the correct prior tokens from the label, not its own guesses."""
    predicted = model_predict_fn(prompt)   # what the (untrained) model currently outputs
    true_dist = {labeled_completion: 1.0}   # 100% probability mass on the ONE correct answer
    pred_dist = {predicted: 0.6, labeled_completion: 0.4}  # toy stand-in distribution
    loss = cross_entropy(true_dist, pred_dist)
    return loss   # weights are then adjusted (via backprop) to shrink this loss

def toy_model(prompt):
    return "I'm not sure."   # stand-in for an untrained model's current guess

for example in sft_dataset:
    loss = sft_training_step(example["prompt"], example["completion"], toy_model)
    print(f"prompt={example['prompt']!r}  loss={loss:.3f}")