Quantified Diversity Gains: The Numbers Behind Verbalized Sampling

Verbalized sampling's value isn't just conceptual — experiments across various tasks report concrete, measurable diversity gains.

The headline number: verbalized sampling significantly enhances output diversity by 1.6-2.1x over direct prompting, while maintaining or even improving output quality — meaning this isn't a diversity-for-quality trade-off, it's closer to a free lunch. Variants of the technique push this further: verbalized-sampling-based Chain-of-Thought and verbalized-sampling-based Multi-turn prompting both improve generation diversity even more than the base VS technique alone.

Interestingly, the benefit scales with model capability rather than shrinking: larger, more capable models like GPT-4.1 and Gemini-2.5-Pro benefit more from verbalized sampling than smaller models do, showing diversity gains up to 2x greater than what smaller models achieve with the same technique — likely because larger models have a richer pre-trained distribution to recover in the first place.

VS also holds up across the training pipeline: it better retains diversity across post-training stages (SFT, DPO, RLVR) than direct prompting does, recovering about 66.8% of the base model's original diversity — compared to a much lower retention rate when using direct prompting on the same aligned model. And critically, VS's gains are independent of other diversity techniques: it composes with methods like temperature scaling and top-p sampling rather than competing with them, so you can stack VS with your existing sampling configuration for further improvements in the diversity-quality trade-off, rather than choosing one or the other.

# A simple distinct-n diversity metric to compare direct vs VS outputs —
# a lightweight way to quantify the diversity gain locally.

def distinct_n(responses: list[str], n: int = 2) -> float:
    """Fraction of unique n-grams across all responses — higher = more diverse."""
    all_ngrams, unique_ngrams = [], set()
    for r in responses:
        words = r.lower().split()
        ngrams = [tuple(words[i:i + n]) for i in range(len(words) - n + 1)]
        all_ngrams.extend(ngrams)
        unique_ngrams.update(ngrams)
    return len(unique_ngrams) / max(1, len(all_ngrams))

direct_outputs = direct_prompt_sample(n=5)                 # from previous subtopic
vs_outputs = [c["response"] for c in verbalized_sampling("Tell me a joke.")]

print("Direct prompting distinct-2:", distinct_n(direct_outputs))
print("Verbalized sampling distinct-2:", distinct_n(vs_outputs))
# Expect the VS score to come out meaningfully higher