Co-Distillation: Training Teacher and Student Together
~12 min read
Instead of a fixed, already-trained teacher, co-distillation starts with BOTH an untrained teacher and an untrained student, training them simultaneously — what Llama 4 used to train Scout and Maverick from Behemoth.
Co-Distillation: Training Teacher and Student Together is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Co-distillation starts with an UNTRAINED teacher AND an untrained student, training both simultaneously, not a fixed already-trained teacher
- •The teacher trains normally on ground-truth hard labels; the student trains to match the teacher's (evolving) soft probabilities
- •Bootstrapping problem: early in training, the teacher's soft labels aren't accurate yet
- •Fix: the student trains on BOTH the teacher's soft labels AND ground-truth hard labels simultaneously, not purely on the teacher's guidance
- •Llama 4 used co-distillation to train Scout and Maverick from Behemoth — a pre-training-stage technique, unlike hard-label distillation's post-training use