Model Pruning and Distillation
~13 min read
Pruning removes weights the model barely uses; distillation trains a small 'student' model to imitate a large 'teacher.' Both shrink the model, but distillation changes the architecture while pruning thins the existing one.
Model Pruning and Distillation is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Pruning removes low-contribution parameters from an existing model; distillation trains a new small student to imitate a large teacher
- •Unstructured pruning zeroes scattered individual weights (saves storage, but standard GPUs rarely run the sparsity faster)
- •Structured pruning removes whole heads/neurons/layers, yielding a smaller dense model that actually runs faster — usually with a recovery fine-tune
- •Distillation's power comes from soft labels: the student learns the teacher's full probability distribution, not just the single correct answer
- •Use structured pruning for a quick faster variant; use distillation for the biggest reduction or to compress a teacher into a model you control — they compose with quantization