Hard-Label Distillation: Solving the Memory Blowup
~12 min read
Instead of the teacher's full probability distribution, only its single final output token gets used — dramatically cheaper to store, at the cost of some of soft-label distillation's richer signal. This is what DeepSeek used to distill DeepSeek-R1.
Hard-Label Distillation: Solving the Memory Blowup is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Hard-label distillation uses only the teacher's final one-hot output token, not its full probability distribution
- •Storage cost drops dramatically — by roughly the vocabulary size — compared to soft-label distillation
- •This is what makes distillation practical at real training-corpus scale, where soft-label storage was infeasible
- •The trade-off: hard labels can't communicate the teacher's confidence or which alternatives it considered plausible
- •DeepSeek used this exact technique to distill DeepSeek-R1's knowledge into Qwen and Llama 3.1 models