Mode Collapse & Typicality Bias: Why Aligned LLMs Lose Diversity
~15 min read
Post-training alignment methods like RLHF make LLMs helpful and safe — but they unintentionally cause mode collapse, where the model starts favoring a narrow set of predictable responses. The root cause is a hidden flaw in the human preference data called typicality bias.
Mode Collapse & Typicality Bias: Why Aligned LLMs Lose Diversity is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Mode collapse: RLHF-aligned LLMs favor a narrow set of predictable responses instead of their full learned diversity
- •Root cause: typicality bias — human annotators naturally favor familiar, predictable responses over equally-good creative ones
- •The reward model learns to mimic this bias, systematically boosting already-likely responses and sharpening the output distribution
- •This isn't irreversible — the pre-trained model's rich diversity still exists in the weights, just suppressed by alignment
- •This suppressed-not-destroyed diversity is exactly what verbalized sampling is designed to recover, without retraining anything