Sampling & Temperature: Controlling Randomness
~10 min read
How an LLM picks its next token is controlled by sampling parameters. Temperature, top-p, and top-k determine whether output is deterministic or creative.
After the forward pass produces logits (raw scores for each vocabulary token), sampling parameters control token selection. Temperature scales logits before softmax — temperature=0 always picks the highest-probability token (greedy, deterministic). Temperature=1 samples from the raw distribution. Temperature=2 makes everything more random and often incoherent.
Top-p (nucleus) sampling: sample from the smallest set of tokens whose cumulative probability exceeds p. Top-p=0.9 means: find the fewest tokens that together have 90% probability, sample from only those. Adapts dynamically — confident predictions use few tokens; uncertain ones use many.
💻 Code example
# Sampling comparison
temperature = 0.0 # deterministic — always highest prob token
temperature = 0.7 # balanced — good for most chat tasks
temperature = 1.0 # creative — sample from full distribution
temperature = 1.5 # very random — often incoherent
top_p = 0.9 # nucleus sampling — prunes low-probability tokens
top_k = 50 # keep only top 50 tokens by probability
# Rule of thumb:
# Factual tasks (Q&A, code): temp=0.1-0.3
# Conversational: temp=0.5-0.7
# Creative writing: temp=0.8-1.0
💬 Deep Dive with AI
Key points
- •Temperature=0: deterministic greedy decoding (same output every time)
- •Temperature>1: more random, often less coherent
- •Top-p=0.9 is a good default — prunes improbable tokens while allowing variation
- •For production Q&A systems, use temperature=0.1-0.3 for consistency