Mode Collapse & Typicality Bias: Why Aligned LLMs Lose Diversity

RLHF and similar post-training alignment methods are designed to make LLMs more helpful and safer to use. But they have a well-documented side effect: a significant drop in output diversity, known as mode collapse. A model that has collapsed to a mode starts favoring a narrow set of predictable or stereotypical responses over the full range of outputs it's actually capable of — ask it for a joke five times and you'll likely get the same joke, or close variants of it, every time.

The root cause, per research into this phenomenon, is a hidden flaw in the human preference data used to train the reward model: typicality bias. Here's the mechanism: annotators rating pairs of LLM responses naturally tend to favor answers that are more familiar, easier to read, and more predictable — even when a more creative, less common answer is equally good or equally correct. This preference isn't malicious or intentional; it's just a natural human bias toward the familiar. But because the reward model is trained to mimic these annotator preferences, it ends up systematically boosting responses that the original (pre-alignment) model already considered likely — which aggressively sharpens the LLM's output distribution down to one or two dominant, highly predictable responses.

The good news: this isn't an irreversible loss of capability. The aligned model effectively has two personalities layered on top of each other — the original model that learned the full rich distribution of possible responses during pre-training, and the safety-focused, post-aligned model that has been unintentionally suppressed toward always favoring the single most predictable response due to typicality bias. The original diversity is still in there, in the model's weights — it's just being systematically suppressed by the alignment process. This is exactly the insight that motivates verbalized sampling: if the diversity is still there, a clever enough prompt should be able to recover it without touching the model's weights at all.

# Illustrating mode collapse: direct prompting tends to converge on
# the same (or near-identical) response across repeated calls.
from openai import OpenAI

client = OpenAI()

def direct_prompt_sample(n: int = 5) -> list[str]:
    results = []
    for _ in range(n):
        resp = client.chat.completions.create(
            model="gpt-4.1",
            messages=[{"role": "user", "content": "Tell me a joke."}],
            temperature=1.0,
        )
        results.append(resp.choices[0].message.content)
    return results

# Even with temperature=1.0, mode collapse means these 5 outputs
# often cluster around the same 1-2 "safe", most-reinforced jokes
# — not the diverse range the base pre-trained model could produce.