Soft-Label Distillation: Maximum Knowledge Transfer (and Its Memory Problem)
~15 min read
The teacher LLM's full softmax probability distribution over the entire vocabulary is used to train the student — maximum knowledge transfer, but a genuinely enormous memory cost to store.
Soft-Label Distillation: Maximum Knowledge Transfer (and Its Memory Problem) is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Soft-label distillation trains the student to match the teacher's FULL softmax distribution, not just its top prediction
- •This full distribution reveals how the teacher reasons — which alternatives it considered plausible and by how much
- •Requires access to the teacher's weights, ruling this technique out for closed-weight, API-only teacher models
- •The memory cost is genuinely enormous: ~500 million GB to store soft labels for a 100k-vocab, 5-trillion-token corpus at float8
- •This memory problem is exactly what motivates hard-label distillation, the next technique