Stage 2 — Instruction Fine-Tuning: Teaching the Model to Follow Prompts

A pre-trained model has absorbed grammar and world knowledge, but it doesn't know how to be helpful in a conversational sense — prompted with a question, it might just continue generating more questions in the same style rather than answering. Instruction fine-tuning (IFT) is the stage that fixes this.

IFT trains the model on curated instruction-response pairs: examples that pair a realistic instruction ('Summarize this article', 'Write a function that reverses a string') with a high-quality response. Instead of the open-ended next-token objective from pre-training, the model is now specifically taught how to follow a prompt and how to format a reply — the difference between 'completing text' and 'responding to a request.'

After IFT, the model can meaningfully answer questions, summarize content, and write code when asked — capabilities that weren't reliably accessible right after pre-training, even though the underlying knowledge was already there. At this point, though, a natural bottleneck appears: the model has now likely been trained on most of the readily available internet text and as much human-labeled instruction-response data as is practical to collect. Getting further improvement from 'more of the same data' has diminishing returns — which is exactly the motivation for the next stage, preference fine-tuning, which uses reinforcement learning instead of more labeled examples.

# Instruction fine-tuning with Hugging Face's trl library —
# SFTTrainer expects a dataset of instruction/response pairs.
from datasets import Dataset
from trl import SFTTrainer, SFTConfig

instruction_pairs = [
    {"prompt": "Summarize: The stock market fell 2% today...",
     "completion": "Markets declined 2% amid rate concerns."},
    {"prompt": "Write a function that reverses a string.",
     "completion": "def reverse_string(s: str) -> str:\n    return s[::-1]"},
    # ...thousands more instruction/response examples
]
dataset = Dataset.from_list(instruction_pairs)

trainer = SFTTrainer(
    model="meta-llama/Llama-3.1-8B",  # a pre-trained base model
    train_dataset=dataset,
    args=SFTConfig(output_dir="./ift-checkpoint", num_train_epochs=3),
)
trainer.train()  # the model learns to follow instructions, not just complete text