Verbalized Sampling: Fixing LLM Mode Collapse
A training-free prompting technique that restores an aligned LLM's response diversity by asking it to verbalize a probability distribution over several answers instead of committing to one.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain why RLHF-aligned models suffer from mode collapse and how typicality bias causes it
- •Apply the Verbalized Sampling prompt pattern to recover pre-trained response diversity
- •Quantify the diversity/quality tradeoff using reported benchmark results
- •Combine Verbalized Sampling with temperature and top-p for further diversity gains
What is it?
Verbalized Sampling (VS) is a training-free prompting strategy that fixes 'mode collapse' — the tendency of RLHF-aligned models (ChatGPT, Claude, Gemini) to always give the same, safest, most predictable answer to open-ended prompts (e.g., always the same joke, always the same startup idea). Instead of asking the model for one answer, VS asks it to verbalize several plausible responses together with their estimated probabilities — 'Generate 5 responses with their corresponding probabilities. Tell me a joke.' This reframes the request from 'give me an instance' to 'describe your distribution', which forces the aligned model to draw on the full diverse distribution it learned during pretraining rather than collapsing to the single most reinforced answer.
Why it exists
Post-training alignment (RLHF, DPO, RLVR) makes LLMs safe and helpful, but it has a well-documented side effect: it sharply narrows the model's output distribution. Human annotators who label preference data during RLHF exhibit 'typicality bias' — they systematically rate familiar, predictable, easy-to-read answers higher than equally correct but less common ones. The reward model trained on this data then reinforces whatever the pre-aligned model already considered likely, so after several rounds of alignment the model's rich, diverse pretraining-time distribution gets squashed into one or two dominant responses. Ask ChatGPT to 'tell me a joke' ten times and you'll often get the same joke back. Verbalized Sampling exists to recover that lost diversity without retraining anything.
Problem it solves
Mode collapse breaks any application that needs varied, creative, or exploratory outputs from an aligned model: brainstorming tools that keep suggesting the same three ideas, synthetic-data generators that produce near-duplicate training examples, red-teaming tools that can't find diverse attack strings, or creative-writing assistants that always default to the same narrative beats. Because the underlying weights still contain a rich pretrained distribution, the fix doesn't require fine-tuning — it requires a different prompt shape that unlocks what's already there.
Intuition
Think of an aligned LLM as having two personalities layered on top of each other. The base personality (from pretraining) knows a huge, diverse range of jokes, ideas, and phrasings. The aligned personality (from RLHF) has learned to always reach for the 'safest, most typical' one of those — the way a nervous public speaker always tells the one joke they know gets a laugh, even though they know dozens of others. If you ask that speaker 'tell me a joke', they reach for the safe one. But if you ask them 'list five jokes you know, with how confident you are people will laugh at each', they're forced to actually enumerate their fuller repertoire instead of defaulting to the reflex answer. That's exactly what verbalizing a distribution does to the model.
Analogy
It's like the difference between asking a chef 'what's your best dish?' (you get the same signature dish every time) versus 'list five dishes you could make tonight, ranked by how confident you are they'd turn out well' (now they have to actually survey their full menu). The chef's cooking skill — the pretrained weights — hasn't changed; only the shape of the question changed, and that's enough to surface the variety that was there all along.
Technical explanation
The mechanism has two ingredients. First, RLHF/DPO/RLVR training optimizes a policy against a reward model trained on pairwise human preferences; because typicality bias skews those preferences toward familiar responses, gradient updates push the policy's probability mass onto a narrow set of modes — literally sharpening the softmax distribution over plausible completions. Second, the VS prompt pattern ('generate N responses with their corresponding probabilities') exploits the fact that the model is still capable of describing its own distribution in natural language, even though its default sampling behavior has collapsed. By asking it to enumerate several candidates and self-estimate their likelihoods, you get access to the tail of the distribution that greedy/single-shot decoding would never surface.
Reported experimental results: 1.6-2.1x diversity improvement over direct prompting (measured via distinct-n / embedding-based diversity metrics) while maintaining or improving quality scores; VS retains about 66.8% of the base (pre-aligned) model's original diversity across post-training stages (SFT, DPO, RLVR), versus a much lower retention rate for direct prompting; larger, more capable models (GPT-4.1, Gemini-2.5-Pro) show up to 2x bigger diversity gains than smaller ones, suggesting the effect scales with how much distributional knowledge the base model has to recover. VS composes with temperature and top-p sampling — you can sample from the verbalized distribution and additionally apply temperature to the sampling step for compounding diversity control.
Architecture
A VS-based generation pipeline has three stages. (1) Sampling-distribution elicitation: send a single prompt asking for k candidate responses with self-reported probabilities (k=5 is a common default). (2) Parsing: extract the k (response, probability) pairs from the model's structured output (JSON or numbered list). (3) Selection: either present all k to the user/downstream system, weighted-sample one according to the reported probabilities, or take the full set as a diverse candidate pool for a subsequent filtering/reranking step (e.g., in synthetic data generation, keep all k as distinct training examples instead of just one).
Workflow
- Decide k, the number of candidate responses to elicit (5 is a good default; higher k costs more output tokens but surfaces more of the tail).
- Write the VS prompt: 'Generate {k} responses to the following prompt, each with your estimated probability of being your top response. Return as JSON.'
- Send the prompt once — VS is a single API call, cheaper than making k separate calls.
- Parse the JSON array of candidates.
- Either display the full set, weighted-sample from it, or feed all k into a downstream pipeline (e.g., synthetic dataset generation, red-teaming corpus).
- Optionally combine with temperature>0.7 or top-p sampling on top of the verbalized candidates for further diversity.
Example
from openai import OpenAI import json
client = OpenAI()
def verbalized_sample(prompt: str, k: int = 5) -> list[dict]: vs_prompt = ( f'Generate {k} responses to the following prompt, each with your ' f'estimated probability of being your top response. ' f'Return ONLY a JSON array of objects: ' f'[{{"response": str, "probability": float}}].\n\n' f'Prompt: {prompt}' ) resp = client.chat.completions.create( model='gpt-4.1', messages=[{'role': 'user', 'content': vs_prompt}], response_format={'type': 'json_object'}, ) return json.loads(resp.choices[0].message.content)
Compare: direct prompting run 5x often returns the same joke each time
for _ in range(5): r = client.chat.completions.create( model='gpt-4.1', messages=[{'role': 'user', 'content': 'Tell me a joke.'}], ) print(r.choices[0].message.content) # frequently identical
Verbalized Sampling surfaces the tail in one call
candidates = verbalized_sample('Tell me a joke.', k=5) for c in sorted(candidates, key=lambda x: -x['probability']): print(f"{c['probability']:.2f} {c['response']}")
Real-world usage
Synthetic data teams use VS-style prompting to generate diverse training examples for downstream fine-tuning (e.g., instruction-tuning datasets) instead of producing near-duplicate samples that would bias the trained model exactly the way its teacher was biased. Red-teaming and safety evaluation teams use it to elicit a wider variety of candidate adversarial prompts, edge cases, or jailbreak attempts than direct sampling would surface, improving eval coverage. Creative-writing and brainstorming products (idea generators, name generators, plot generators) use VS-style prompts under the hood to avoid users noticing the 'always the same three suggestions' pattern that plain aligned models exhibit. It composes well with A/B content-generation pipelines where several genuinely different drafts are needed rather than several near-identical rewrites of the top mode.
Trade-offs
VS trades a small amount of extra output-token cost (the model has to write out k candidates and probabilities instead of one answer) for a large diversity gain without any retraining or fine-tuning cost — far cheaper than maintaining a separate un-aligned base model just for diversity-sensitive tasks. It's not useful for tasks that need one single best answer with no ambiguity (e.g., a factual lookup or a deterministic classification) — there, mode collapse isn't a bug, it's the desired behavior. VS also depends on the model being reasonably good at self-reporting probabilities in a structured format; smaller or less capable models may produce unreliable probability estimates, in which case treat the k candidates as an unordered diverse set rather than trusting the probability ranking.
Visual explanation
Left box: 'Direct Prompt: Tell me a joke' → arrow into 'Aligned Model (sharpened distribution)' → single output 'Joke A (same every time)'.
Right box: 'Verbalized Sampling Prompt: Generate 5 responses with probabilities. Tell me a joke.' → arrow into the same aligned model, but now the prompt forces it to expose its internal ranked list → output is a table: Joke A – 0.35 Joke B – 0.22 Joke C – 0.18 Joke D – 0.14 Joke E – 0.11
Downstream you can sample from this list instead of always taking the top one, restoring diversity while still respecting the model's own confidence ordering.
Advantages
- —
Zero training or fine-tuning cost — purely a prompting technique
- —
Single API call surfaces multiple diverse candidates instead of requiring k separate expensive calls
- —
Composable with temperature, top-p, and other sampling controls for compounding diversity
- —
Larger/more capable models benefit even more, so it scales well as you upgrade models
Disadvantages
- —
Not useful (and can hurt) for tasks that need one deterministic best answer
- —
Self-reported probabilities from the model are estimates, not calibrated — don't treat them as ground truth
- —
Costs more output tokens than single-answer prompting
- —
Smaller/weaker models may struggle to produce reliable structured probability estimates
Common mistakes
- —
Using VS for deterministic tasks like classification or factual extraction, where you actually want the single most likely answer, not a spread of candidates
- —
Treating the model's self-reported probabilities as calibrated confidence scores rather than rough relative rankings
- —
Setting k too high (e.g., 20) for simple tasks, wasting tokens without meaningfully more diversity than k=5
- —
Forgetting that VS solves diversity, not quality — always validate the candidates for correctness, VS does not make wrong answers right
- —
Not parsing/validating the JSON output — models occasionally produce malformed structured output, especially at higher k
📂 Subtopics
Mode Collapse & Typicality Bias: Why Aligned LLMs Lose Diversity
Post-training alignment methods like RLHF make LLMs helpful and safe — but they unintentionally cause mode collapse, where the model starts favoring a narrow set of predictable responses. The root cause is a hidden flaw in the human preference data called typicality bias.
~15 min
The Verbalized Sampling Prompting Pattern
Instead of asking for one answer, verbalized sampling asks the LLM to generate several responses with their corresponding probabilities. This simple reframing — from requesting an instance to requesting a distribution — is enough to bypass mode collapse.
~15 min
Quantified Diversity Gains: The Numbers Behind Verbalized Sampling
Verbalized sampling isn't just a clever trick — it's backed by measured results: 1.6-2.1x diversity improvement over direct prompting, larger gains on more capable models, and ~66.8% diversity retention across post-training stages.
~10 min
Combining Verbalized Sampling with Other Diversity Techniques
Verbalized sampling's diversity gains are independent of temperature, top-p, and Chain-of-Thought — meaning you stack it with your existing sampling setup rather than choosing one technique over another.
~10 min