SFT vs RFT: Choosing a Fine-Tuning Objective
The decision framework for choosing between supervised fine-tuning (static labeled data) and reinforcement fine-tuning (online reward-based exploration via GRPO) — including the full labeled-data/verifiability decision tree.
Fine-Tuning Controls:
SFT vs. RFT — Decision Tree
Walk the tree from the root: whether you have labeled data, whether the task is automatically verifiable, and how much data you have all determine which fine-tuning objective fits.
SFT vs. RFT — Decision Tree
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Contrast the SFT process (static labeled data, weight matching) with the RFT process (online reward exploration via GRPO)
- •Apply the full decision tree: labeled data? verifiable task? reasoning-helps? to choose SFT, RFT, or RLHF
- •Explain why RFT is well suited to reasoning-heavy tasks like math and logic
- •Identify when a tiny dataset should still use SFT instead of RFT
What is it?
SFT (Supervised Fine-Tuning) and RFT (Reinforcement Fine-Tuning) are the two dominant paradigms for fine-tuning an LLM, and choosing between them is a distinct decision from choosing a specific technique like LoRA. SFT starts with a static labeled dataset of prompt-completion pairs and adjusts model weights to match those completions directly. RFT uses an online reward approach instead — no static labels required — where the model explores different outputs, a reward function scores their correctness, and the model learns over time to generate higher-reward answers, typically using GRPO (Group Relative Policy Optimization).
Why it exists
Before RFT-style methods became practical, SFT was effectively the only fine-tuning objective available, and its limitation is structural: SFT trains a model to match static labeled examples, which means it often ends up memorizing answers rather than learning to reason or generalize to genuinely novel problems. RFT exists to fix this for a specific class of tasks — those where correctness can be automatically verified — by letting the model explore its own outputs and learn from a reward signal rather than being told the exact target completion, which produces genuine reasoning and exploration rather than memorization.
Problem it solves
The core distinction solves the 'which training objective actually matches my task and data situation' problem. SFT uses static data and often memorizes answers — great when you have clean labeled examples and want the model to reliably reproduce a known-good pattern. RFT, being online, learns from rewards and explores new strategies — better suited when correctness can be verified automatically (a math answer is right or wrong; code either passes tests or doesn't) and you want the model to discover strategies beyond what's in any fixed labeled dataset. Explicitly naming both options, and the decision criteria between them, solves the common mistake of defaulting to SFT (the more familiar, longer-established method) even when a task's structure actually favors RFT.
Intuition
SFT is like teaching by showing worked example solutions and having the student memorize the pattern — reliable for reproducing known solution types, but the student may not develop genuine problem-solving ability beyond the examples shown. RFT is like giving the student many practice problems with an answer-checker, letting them try different approaches and learn from which ones actually work — slower to set up (you need a way to automatically check correctness), but it produces a student who has genuinely learned to solve problems, not just recall memorized answers.
Analogy
SFT is like training a chess engine by showing it a database of grandmaster games and having it learn to imitate those exact moves. RFT is like letting the engine play millions of games against itself, learning purely from whether it wins or loses (the reward signal), discovering strategies that may not appear in any human game database at all. The imitation approach (SFT) is faster to bootstrap and works even without a way to 'play out' a game to see who wins; the self-play approach (RFT) can discover genuinely novel, superior strategies but requires a reliable way to score outcomes.
Technical explanation
The SFT process: start with a static labeled dataset of prompt-completion pairs, adjust model weights (typically via a LoRA-style parameter-efficient method) to match these completions, and deploy the best resulting checkpoint for inference. The RFT process: instead of static labels, use an online reward approach — the model generates/explores different candidate outputs for a given prompt, a reward function scores each output's correctness (deterministically, for verifiable tasks), and over time the model learns to generate higher-reward answers, typically using GRPO as the specific reinforcement learning algorithm driving these updates.
The full decision tree: first ask whether you have labeled (ground-truth) data at all. If you don't, ask whether the task is verifiable — can correctness be automatically checked by a deterministic function (e.g., does this code pass its tests, is this the correct numeric answer)? If not verifiable, you need RLHF, since only humans can provide the necessary preference signal for genuinely subjective or unverifiable quality judgments. If verifiable, RFT works well, since correctness can be automatically checked without ever needing human-labeled targets. If you DO have labeled data, the choice depends on how much you have: with a large labeled dataset, use SFT — there's enough signal to directly train on matching completions. With only a tiny labeled dataset, ask whether reasoning (like Chain-of-Thought) actually helps on this task — if yes, prefer RFT (since a small labeled dataset alone won't teach robust reasoning, but a reward-based exploration process can); if reasoning doesn't meaningfully help the task, SFT on the tiny dataset is still the simpler, sufficient choice.
For reasoning-heavy tasks specifically (math, logic), GRPO is one of the most effective RFT methods available precisely because these tasks are naturally verifiable (a math answer is checkably right or wrong) and benefit substantially from exploration-driven learning rather than static example memorization.
Architecture
SFT's architecture is a straightforward supervised training loop: (prompt, completion) pairs → forward pass → loss against the target completion → backpropagation → updated weights (typically via LoRA). RFT's architecture is a reinforcement learning loop: prompt → LLM generates multiple candidate completions → each is scored by a (typically deterministic, verification-based) reward function → rewards are aggregated into a training signal (e.g., via GRPO's group-relative advantage calculation) → a policy-gradient-style loss updates the model → the loop repeats, with the model's own generated outputs (not fixed external labels) driving the learning signal.
Workflow
- Determine whether you have labeled (ground-truth) data for your task at all.
- If you don't have labeled data, determine whether the task is verifiable — is there a deterministic way to check if an output is correct?
- If not verifiable (subjective quality, taste, style), use RLHF, since only human preference signals can supply the needed reward.
- If verifiable (math, code, structured logic), use RFT — you can build a reward function without ever needing human labels.
- If you do have labeled data, check how much you have.
- With a large labeled dataset, use SFT directly.
- With only a tiny labeled dataset, evaluate whether reasoning/CoT-style exploration would meaningfully help this specific task — if yes, prefer RFT despite having some labeled data; if reasoning doesn't help, SFT on the tiny dataset remains the simpler, sufficient choice.
- Once RFT is selected for a reasoning-heavy task, default to GRPO as the specific algorithm, given its effectiveness on math/logic-style verifiable tasks.
Example
def choose_finetuning_objective( has_labeled_data: bool, is_verifiable: bool = None, dataset_size: str = None, # 'large' | 'tiny' reasoning_helps: bool = None, ) -> str: if not has_labeled_data: return 'RFT' if is_verifiable else 'RLHF' if dataset_size == 'large': return 'SFT' # tiny dataset return 'RFT' if reasoning_helps else 'SFT'
Examples:
choose_finetuning_objective(has_labeled_data=False, is_verifiable=True) # 'RFT' (math/code) choose_finetuning_objective(has_labeled_data=False, is_verifiable=False) # 'RLHF' (subjective quality) choose_finetuning_objective(has_labeled_data=True, dataset_size='large') # 'SFT' (ample labels) choose_finetuning_objective(has_labeled_data=True, dataset_size='tiny', reasoning_helps=True) # 'RFT'
Real-world usage
DeepSeek-R1's training pipeline applies GRPO (an RFT method) directly on math and code verification rewards — compiler pass/fail, correct/incorrect numeric answers — requiring no human preference data at all for that stage, a direct real-world application of the 'verifiable task, no labels' branch of the decision tree. Most instruction-tuned chat models (the vast majority of fine-tuned open-weight models on Hugging Face) rely primarily on SFT, since general instruction-following has abundant labeled (prompt, ideal-response) data available and doesn't cleanly reduce to a verifiable reward function. Code-generation and math-tutoring specialized models increasingly incorporate an RFT/GRPO stage on top of SFT specifically because code correctness (tests pass) and math correctness (right answer) are both naturally verifiable, making the RFT branch of the decision tree directly applicable.
Trade-offs
SFT is simpler to implement, requires no reward-function design, and works for any task where you have labeled data — but its static nature means it often ends up memorizing patterns from the training set rather than developing generalizable reasoning, and it requires meaningful labeled-data volume to work well. RFT can produce genuinely better reasoning and generalization by learning from exploration and reward rather than memorized targets, and doesn't require labeled completions at all — but it only works for verifiable tasks (you need a reliable reward function), is more complex to implement (GRPO training loops, reward function design, sampling multiple candidates per prompt), and is typically more compute-intensive per training step than straightforward SFT.
Visual explanation
A decision-tree flowchart. Root: 'Do you have labeled (ground-truth) data?' → No branch: 'Is the task verifiable (can correctness be automatically checked)?' → No: use RLHF (humans provide preference signals). → Yes: use RFT (correctness can be automatically checked). → Yes branch (you DO have labeled data): 'How much labeled data do you have?' → Large dataset: use SFT. → Tiny dataset: 'Does reasoning (like CoT) help on this task?' → Yes: use RFT. → No: use SFT.
Advantages
- —
The decision tree reduces a genuinely confusing choice (SFT vs RFT vs RLHF) to a small number of concrete yes/no questions
- —
RFT enables fine-tuning entirely without labeled data for verifiable tasks, removing a major cost/bottleneck (human labeling) for math/code-style domains
- —
SFT remains simple, well-understood, and sufficient for the (very common) case of large labeled datasets on non-reasoning-heavy tasks
- —
Explicitly naming RLHF as the fallback for unverifiable tasks prevents teams from forcing RFT onto a task it structurally cannot handle
Disadvantages
- —
RFT requires designing a reliable, deterministic reward function, which is nontrivial engineering work for many real tasks
- —
SFT alone tends to memorize training examples rather than developing genuinely transferable reasoning ability
- —
The decision tree's branches (verifiable? large dataset? does reasoning help?) sometimes require real judgment calls rather than clean yes/no answers
- —
RFT/GRPO training loops are more compute-intensive and operationally complex than straightforward SFT
Common mistakes
- —
Defaulting to SFT out of familiarity even when a task is naturally verifiable and would benefit more from RFT's exploration-driven learning
- —
Attempting RFT on a genuinely unverifiable, subjective task where no reliable reward function can be built — that's specifically what RLHF exists for instead
- —
Using SFT on a tiny dataset for a task where reasoning genuinely helps, when RFT would likely generalize better from the same small amount of signal
- —
Assuming RFT is always superior to SFT regardless of data volume — with a genuinely large labeled dataset, SFT remains the simpler and sufficient choice
- —
Skipping the 'is the task verifiable' question entirely and jumping straight to RFT without a real, tested reward function in place
📂 Subtopics
Supervised Fine-Tuning (SFT): Static Labels, Matching Completions
SFT fine-tunes a model on a fixed dataset of prompt-completion pairs, adjusting weights so the model's output matches the given completions — the book's baseline fine-tuning approach before RFT.
~12 min
Reinforcement Fine-Tuning (RFT): Reward Signals Instead of Static Labels
RFT replaces fixed labeled completions with an online reward signal: the model explores different outputs, a reward function scores correctness, and the model learns to generate higher-reward answers via GRPO.
~13 min
When to Use SFT vs RFT: The Book's Decision Tree
The book's exact decision flowchart: labeled data or not? Verifiable outcome or not? Large or tiny dataset? Each branch points to a specific, concrete choice — SFT, RFT, or RLHF.
~12 min
Practical Tradeoffs: Data, Compute, Stability and Real Examples
Beyond the decision tree, SFT and RFT differ practically in data requirements, compute cost, and training stability — tradeoffs worth understanding even after the tree points you toward one.
~12 min