3 Techniques to Train an LLM Using Another LLM (Distillation)
Soft-label, hard-label, and co-distillation — the three ways real production LLMs (Llama 4, Gemma, DeepSeek) train a smaller student model using a larger teacher.
Fine-Tuning Controls:
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain the difference between pre-training-stage and post-training-stage distillation
- •Contrast soft-label distillation's memory cost problem with hard-label distillation's fix
- •Explain co-distillation's simultaneous teacher-student training process
- •Name real production models trained with each distillation technique
What is it?
LLMs don't just learn from raw text — they also learn from each other, via a technique called distillation, which transfers 'knowledge' from one LLM (the teacher, typically larger) to another (the student, typically smaller). This is a well-established idea from traditional deep learning, now applied at LLM scale. Real production examples: Llama 4 Scout and Maverick were trained using Llama 4 Behemoth as a teacher; Gemma 2 and 3 were trained using Google's proprietary Gemini. There are three commonly used distillation techniques: soft-label distillation (student learns to match the teacher's full probability distribution), hard-label distillation (student learns to match only the teacher's single final output token), and co-distillation (teacher and student train simultaneously, each learning from the other).
Why it exists
Training a large, capable 'teacher' model from scratch is extremely expensive, but once you have one, you often want a smaller, cheaper-to-run 'student' model that captures as much of the teacher's capability as possible without the same inference cost. Distillation exists to transfer that expensively-learned knowledge efficiently — rather than training the smaller model from scratch on raw text alone (which wastes the fact that a much more capable model already exists and could provide richer training signal), the student learns directly from the teacher's own outputs or behavior, typically producing a stronger smaller model than training on raw data alone would.
Problem it solves
Soft-label distillation solves the 'how do we transfer maximum knowledge' problem by giving the student visibility into the teacher's full probability distribution (not just its single best answer), which carries much richer signal — including the teacher's implicit uncertainty and secondary considerations for every token. But soft-label distillation itself has a serious problem: storing the teacher's full softmax probability distribution over a large vocabulary for every token in a large training corpus requires an enormous amount of memory (for a 100K vocabulary and 5 trillion token corpus, even at float8 precision, this works out to roughly 500 million GB). Hard-label distillation solves exactly this memory problem, by having the student learn to match only the teacher's single, final one-hot output token rather than its full probability distribution, at the cost of losing some of that richer distributional signal. Co-distillation solves a different problem: it doesn't require a pre-existing, fully-trained teacher at all — both models train together from an untrained state, useful when you want to develop a family of models (a large one and a small one) simultaneously rather than sequentially.
Intuition
Soft-label distillation is like a master craftsman not just showing an apprentice the final finished piece, but explaining their full reasoning at every step — which options they considered and roughly how strongly they favored each, not just which one they ultimately picked. This is incredibly rich training signal, but writing down and storing that full reasoning for every single step of every single project quickly becomes an overwhelming amount of documentation. Hard-label distillation is the master craftsman only showing the apprentice the final finished piece at each step, without the reasoning — much less to record and store, but the apprentice loses the nuance of the runner-up considerations. Co-distillation is like a master and an apprentice learning a craft together from day one, at the same time, with the apprentice specifically trying to imitate the master's evolving technique as the master's own skill develops in parallel, rather than the master already being a fully-formed expert from the start.
Analogy
Think of soft-label distillation like a teacher grading practice exams not just with a checkmark or an X, but with a full percentage breakdown of how likely each possible answer choice seemed to them ('I was 60% confident in A, 25% in B, 10% in C, 5% in D') — extremely informative for the student, but writing that level of detail for every single question on every single practice exam for an entire semester becomes an enormous amount of paperwork. Hard-label distillation is the same teacher just marking the single correct answer with no breakdown — much less paperwork, but the student loses the nuance of 'the teacher considered B a plausible near-miss.' Co-distillation is like a senior expert and a junior colleague both learning a genuinely new skill together in real time, at the same pace, with the junior specifically trying to track and mirror the senior's evolving approach as they both improve, rather than the senior already being a finished expert on day one.
Technical explanation
Distillation in LLMs can happen at two distinct stages: pre-training (train the bigger teacher LLM and the smaller student LLM together, from the start — Llama 4 did this) or post-training (train the bigger teacher LLM first to completion, then distill its already-learned knowledge into the smaller student LLM afterward — DeepSeek did this, distilling DeepSeek-R1's capabilities into Qwen and Llama 3.1-based student models). Some approaches, like Gemma 3, apply distillation during both stages.
(1) Soft-label distillation: use a fixed, already-pretrained teacher LLM to generate softmax probabilities over the entire training corpus, pass the same data through the untrained student LLM to get its own softmax probabilities, and train the student to match the teacher's probability distribution (not just its top pick). This gives the student visibility into the teacher's full probability distribution, ensuring maximum knowledge and reasoning transfer — but requires access to the teacher's weights to obtain that output distribution, and has a severe storage problem: for a 100K-token vocabulary and a 5-trillion-token training corpus, storing the softmax probability distribution for every input token over the entire vocabulary would require roughly 500 million GB of memory even at float8 precision.
(2) Hard-label distillation: use the fixed, pretrained teacher LLM to produce just the final one-hot output token (its single top choice, not a distribution), and train the student LLM (processing the same data) to match this. This directly solves soft-label distillation's massive storage problem, since you only need to store a single token id per position rather than a full probability vector — at the cost of losing the richer distributional signal that soft-label distillation provides. DeepSeek used this technique to distill DeepSeek-R1's capabilities into Qwen and Llama 3.1 student models.
(3) Co-distillation: start with BOTH an untrained teacher LLM and an untrained student LLM (unlike the previous two techniques, which assume a fixed, already-trained teacher). Generate softmax probabilities over the current training batch from both models simultaneously; train the teacher LLM as usual on the hard ground-truth labels; train the student LLM to match its softmax probabilities to those of the (simultaneously-training) teacher. Because the teacher's own soft labels aren't accurate yet during early training (it hasn't finished learning either), the student is trained using BOTH the teacher's soft labels AND the ground-truth hard labels together, blending the two signals. Llama 4 used co-distillation to train Llama 4 Scout and Maverick from the larger Llama 4 Behemoth model.
Architecture
All three techniques share the same fundamental teacher-student structure but differ in (a) whether the teacher is fixed/pretrained or trained simultaneously with the student, and (b) exactly what signal flows from teacher to student. Soft-label and hard-label distillation both assume a fixed, already-trained teacher and a post-training-style distillation process, differing only in whether the transferred signal is a full probability distribution (soft) or a single token (hard). Co-distillation is architecturally distinct — both models are training simultaneously from scratch, with training signal flowing bidirectionally in the sense that the student tracks the teacher's evolving state in real time, rather than the teacher being a static, finished source of truth.
Workflow
- Decide whether you have (or want to first train) a large, capable teacher model, or whether you want to train teacher and student together from scratch — this determines whether you're doing soft/hard-label distillation (fixed teacher) or co-distillation (simultaneous training).
- If using a fixed, pretrained teacher and storage/memory is not a binding constraint, prefer soft-label distillation for its richer knowledge transfer.
- If using a fixed, pretrained teacher but storage of full probability distributions over your corpus is infeasible (as it typically is at real LLM scale), use hard-label distillation instead.
- If you want to develop a family of models (a large flagship and a smaller efficient variant) simultaneously rather than sequentially, use co-distillation.
- For co-distillation specifically, blend the teacher's (evolving, initially inaccurate) soft labels with the ground-truth hard labels in the student's loss, especially early in training when the teacher's own outputs aren't yet reliable.
- Decide whether to apply distillation at the pre-training stage, the post-training stage, or both (as Gemma 3 did), based on whether you want the student to develop its foundational capabilities under the teacher's guidance from the start, or only inherit the teacher's fully-developed capabilities afterward.
Example
import torch.nn.functional as F
Soft-label distillation loss: KL divergence between full distributions
def soft_label_loss(student_logits, teacher_logits, temperature=2.0): student_probs = F.log_softmax(student_logits / temperature, dim=-1) teacher_probs = F.softmax(teacher_logits / temperature, dim=-1) return F.kl_div(student_probs, teacher_probs, reduction='batchmean')
Hard-label distillation loss: cross-entropy against teacher's top choice
def hard_label_loss(student_logits, teacher_logits): teacher_top_token = teacher_logits.argmax(dim=-1) return F.cross_entropy(student_logits, teacher_top_token)
Co-distillation loss: blend teacher-soft-label signal with ground truth
def co_distillation_loss(student_logits, teacher_logits, ground_truth, alpha=0.5): soft_loss = soft_label_loss(student_logits, teacher_logits) hard_loss = F.cross_entropy(student_logits, ground_truth) return alpha * soft_loss + (1 - alpha) * hard_loss # blend both signals
Real-world usage
Llama 4 Scout and Maverick (Meta) were trained using Llama 4 Behemoth as the teacher via co-distillation. Gemma 2 and Gemma 3 (Google) were trained using Google's proprietary Gemini as the teacher. DeepSeek distilled DeepSeek-R1's reasoning capabilities into smaller Qwen and Llama 3.1-based student models using hard-label distillation, producing a family of smaller, more efficient models that retain much of DeepSeek-R1's reasoning strength at a fraction of the inference cost. Gemma 3 specifically applied distillation at both the pre-training and post-training stages, combining the benefits of both approaches.
Trade-offs
Soft-label distillation provides the richest knowledge transfer but is often practically infeasible at real LLM scale due to its enormous storage requirements for full probability distributions across a large vocabulary and corpus. Hard-label distillation is far more practical to store and compute but loses the nuanced distributional signal (the teacher's implicit 'runner-up' considerations for each token) that soft-label distillation provides. Co-distillation doesn't require a pre-existing trained teacher at all, letting you develop a model family simultaneously — but during early training, the teacher's own soft labels are unreliable, requiring the student to also be trained on hard ground-truth labels to compensate, adding some complexity to the loss design.
Visual explanation
Three side-by-side pipeline diagrams.
Soft-label distillation: [Fixed Pre-trained Teacher LLM] → generates [full softmax probability distribution over the corpus] → [Untrained Student LLM] processes the same data → generates its own softmax distribution → [Loss: match Student's distribution to Teacher's distribution] → gradient updates flow only to the Student.
Hard-label distillation: same pipeline, but the Teacher only outputs a [single one-hot final token] rather than a full distribution, dramatically shrinking the stored/transmitted data volume.
Co-distillation: [Untrained Teacher LLM] and [Untrained Student LLM] both process the current batch simultaneously → Teacher trains normally on hard ground-truth labels → Student's loss combines matching the Teacher's (evolving) softmax output AND the ground-truth hard labels directly, shown as two arrows feeding into the Student's loss.
Advantages
- —
Distillation transfers expensively-learned knowledge from a large teacher to a smaller, cheaper-to-run student far more efficiently than training the student on raw text alone
- —
Hard-label distillation makes distillation practically feasible at real LLM scale by avoiding soft-label distillation's prohibitive storage requirements
- —
Co-distillation lets teams develop an entire model family (large + small variants) simultaneously rather than sequentially, without waiting for a teacher to finish training first
- —
Real production models (Llama 4, Gemma, DeepSeek-distilled Qwen/Llama variants) validate all three techniques work at genuine LLM scale
Disadvantages
- —
Soft-label distillation's storage requirements are prohibitive at real LLM vocabulary and corpus scale, limiting its practical use despite its richer signal
- —
Hard-label distillation loses the nuanced distributional information that makes soft-label distillation theoretically superior
- —
Co-distillation's early-training unreliable teacher signal adds loss-design complexity (blending soft labels with hard ground-truth) that fixed-teacher approaches don't need
- —
All three techniques still require access to a capable teacher model (either pretrained or trained alongside), which not every team has
Common mistakes
- —
Attempting soft-label distillation at real LLM vocabulary/corpus scale without first checking the storage requirements, which can be genuinely infeasible (hundreds of millions of GB)
- —
Using hard-label distillation when soft-label distillation's storage cost was actually manageable for the specific (smaller-scale) use case, unnecessarily sacrificing signal richness
- —
Attempting co-distillation without blending in ground-truth hard labels for the student, ignoring that the teacher's own early-training soft labels are unreliable
- —
Confusing pre-training-stage distillation (training teacher and student together from the start) with post-training-stage distillation (distilling from an already-finished teacher) — they have different infrastructure and timing requirements
- —
Assuming distillation requires the teacher and student to share the same architecture — production examples (DeepSeek-R1 → Qwen/Llama) demonstrate cross-architecture distillation works
📂 Subtopics
Soft-Label Distillation: Maximum Knowledge Transfer (and Its Memory Problem)
The teacher LLM's full softmax probability distribution over the entire vocabulary is used to train the student — maximum knowledge transfer, but a genuinely enormous memory cost to store.
~15 min
Hard-Label Distillation: Solving the Memory Blowup
Instead of the teacher's full probability distribution, only its single final output token gets used — dramatically cheaper to store, at the cost of some of soft-label distillation's richer signal. This is what DeepSeek used to distill DeepSeek-R1.
~12 min
Co-Distillation: Training Teacher and Student Together
Instead of a fixed, already-trained teacher, co-distillation starts with BOTH an untrained teacher and an untrained student, training them simultaneously — what Llama 4 used to train Scout and Maverick from Behemoth.
~12 min
When to Use Which: Pre-training vs. Post-training Distillation Choices
The right distillation technique depends on whether you have weight access to the teacher, your memory budget, and whether you're distilling during pre-training or post-training — a concrete decision guide across all 3 techniques.
~12 min