Combining Verbalized Sampling with Other Diversity Techniques

A natural question once you understand verbalized sampling: does it replace temperature and top-p sampling, or work alongside them? The answer is the latter — VS's diversity gains are independent of these other techniques, meaning you get to compose them rather than pick one.

Temperature and top-p operate at the token-sampling level, reshaping the probability distribution the model samples from at each individual generation step. Verbalized sampling operates at the prompt level — it changes what you're asking the model to produce (a distribution of responses, not a single instance) before any token-level sampling even happens. Because these two things act at different layers of the generation process, stacking them compounds their effects rather than one canceling the other out: you can run verbalized sampling at temperature=1.0 with top-p=0.9, getting both VS's distribution-level diversity recovery and the token-level diversity that a well-tuned temperature/top-p setting already provides.

The same composability holds for VS-based Chain-of-Thought and VS-based Multi-turn prompting — these aren't separate, competing techniques, they're the verbalized-sampling pattern applied inside reasoning chains or multi-turn conversations respectively, and both push diversity gains higher than base VS alone by combining the distribution-request framing with the specific structure of CoT or multi-turn dialogue.

The practical takeaway: if you're already using temperature/top-p tuning to fight repetitive output and it isn't fully solving the problem (especially on heavily-aligned models), verbalized sampling is a genuinely additive lever to pull, not a replacement for what you're already doing.

# Stacking verbalized sampling with temperature + top-p — the two
# operate at different layers, so their effects compound.
from openai import OpenAI
import json

client = OpenAI()

def verbalized_sampling_stacked(
    question: str, k: int = 5, temperature: float = 1.0, top_p: float = 0.9,
) -> list[dict]:
    vs_prompt = (
        f"Generate {k} responses with their corresponding probabilities, "
        f"sampled from the full distribution. Format as a JSON list of "
        f"objects with 'response' and 'probability' fields.\n\n{question}"
    )
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": vs_prompt}],
        temperature=temperature,  # token-level diversity lever
        top_p=top_p,              # token-level diversity lever
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)["responses"]
    # VS (prompt-level) + temperature/top-p (token-level) compound —
    # neither one alone recovers as much diversity as both together