When to Use Which: Pre-training vs. Post-training Distillation Choices
~12 min read
The right distillation technique depends on whether you have weight access to the teacher, your memory budget, and whether you're distilling during pre-training or post-training — a concrete decision guide across all 3 techniques.
When to Use Which: Pre-training vs. Post-training Distillation Choices is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Pre-training-stage distillation (teacher + student trained together) → co-distillation, as Llama 4 did for Scout/Maverick
- •Post-training-stage distillation (teacher already finished) → soft-label or hard-label, as DeepSeek did for R1-to-Qwen/Llama
- •You can combine both stages, as Gemma 3 did
- •Within post-training: soft-label needs teacher weight access AND a feasible memory budget for full distributions
- •No weight access, or an infeasible memory budget (very common at real corpus scale) → hard-label distillation