Text Generation: Decoding & Sampling
Learn generation parameters (temperature, top-p, top-k) and local execution runtimes.
KV Parameters:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Explain the difference between Greedy Search and Sampling decoding
- •Analyze how Temperature adjusts probability curves
- •Compare Top-P (nucleus) and Top-K sampling boundaries
What is it?
Text generation is the autoregressive process by which a trained LLM produces output one token at a time. At each step, the model runs a forward pass over all previous tokens and outputs a probability distribution over the entire vocabulary (logits → softmax). The decoding strategy determines how to sample from that distribution: greedy (argmax), beam search (top-k paths), temperature sampling, nucleus (top-p) sampling, or min-p sampling. These strategies trade off diversity vs. coherence vs. speed.
Why it exists
If we always pick the absolute most likely token (Greedy search), the model gets stuck in repetitive loops and generates dry text.
Problem it solves
Solves repetitive phrasing, lack of creativity, model loops, and raw local execution latency.
Intuition
If you want to pick what to eat, greedy search is ordering the same favorite dish every day. Sampling is rolling a dice between a few top options to add variety.
Analogy
Temperature is like a stove dial: low temperature keeps the molecules (probabilities) frozen and orderly. High temperature makes them boil and bounce around randomly.
Technical explanation
Given context x_{1:t}, the model outputs logits z ∈ ℝ^|V|. The sampling pipeline:
- Temperature scaling: z'_i = z_i / T. T→0 collapses to greedy; T>1 flattens the distribution.
- Top-K filtering: keep only the K highest-probability tokens, set rest to −∞ before softmax.
- Top-P (nucleus) filtering: sort by probability descending; keep the smallest set whose cumulative probability ≥ p. Adapts to the distribution shape — narrow when the model is confident, broad when uncertain.
- Min-P: filters tokens whose probability < min_p × p_max. Clips the long tail dynamically.
- Sample token id ~ Categorical(softmax(z')). Append to context. Repeat.
Beam search maintains B candidate sequences, expanding each by the top-B tokens and pruning to keep the B highest cumulative log-probabilities. More deterministic than sampling but prone to length bias and generic outputs. Rarely used in practice for chat models.
KV-cache: the attention Key/Value matrices for all previous tokens are cached. Each new token only runs a single-row attention computation, reducing inference cost from O(n²) per step to O(n) per step. Without KV-cache, generating 1000 tokens would require 1000 full forward passes.
Architecture
Inference stack components: Logit processor: applies temperature, top-k, top-p, repetition penalty in sequence. Sampling controller: draws from the filtered distribution (torch.multinomial or equivalent). KV-cache manager: allocates and pages GPU memory for attention keys/values (PagedAttention in vLLM). Continuous batching: processes multiple requests simultaneously by dynamically adding/removing sequences from a shared batch — avoids GPU idle time between requests. Speculative decoding: a small draft model generates K tokens; the large model verifies them in a single parallel forward pass, accepting or rejecting. Achieves 2–3× throughput at zero quality cost. Local runtimes: llama.cpp (GGUF quantized weights, CPU+GPU), Ollama (wrapper), vLLM (production server with PagedAttention), TGI (HuggingFace), SGLang.
Workflow
- Get logits -> 2. Scale by Temperature -> 3. Apply Top-K -> 4. Apply Top-P -> 5. Sample token ID.
Example
Sampling pipeline — shows exactly what happens inside generate()
import torch, torch.nn.functional as F
def sample_next_token(logits: torch.Tensor, # (vocab_size,) temperature: float = 1.0, top_k: int = 50, top_p: float = 0.9) -> int: # 1. Temperature logits = logits / max(temperature, 1e-8)
# 2. Top-K
if top_k > 0:
kth = torch.topk(logits, top_k).values[-1]
logits[logits < kth] = float('-inf')
# 3. Top-P (nucleus)
probs = F.softmax(logits, dim=-1)
sorted_p, sorted_idx = torch.sort(probs, descending=True)
cumsum = sorted_p.cumsum(dim=-1)
remove = cumsum - sorted_p > top_p
sorted_p[remove] = 0.0
sorted_p /= sorted_p.sum() # renormalize
# 4. Sample
return sorted_idx[torch.multinomial(sorted_p, 1)].item()
Typical chat config: temperature=0.7, top_p=0.9, top_k=0 (no top-k)
Code generation: temperature=0.0 (greedy/deterministic)
Creative writing: temperature=1.2, top_p=0.95
Real-world usage
OpenAI's API exposes temperature, top_p, and frequency_penalty. Setting temperature=0 is idempotent — produces the same output for the same input, essential for structured JSON extraction and deterministic pipelines. vLLM in production uses PagedAttention to serve 100+ concurrent requests on a single A100: KV-cache memory is paged like virtual memory, eliminating fragmentation. Speculative decoding (llama.cpp -draft flag): a 70B model uses a 7B draft model — draft proposes 4 tokens, main model accepts 3.2 on average → ~3× throughput improvement. Repetition penalty (Ollama repeat_penalty): multiplies logits of recently seen tokens by a factor <1 to prevent loops — critical when temperature is high or context is long.
Trade-offs
Greedy search (temperature=0) is highly logical but repetitive; high temperature is creative but erratic.
Visual explanation
Top-K vs. Top-P boundaries: Top-K: Keep exactly the top 5 tokens, discard others. Top-P: Keep tokens until their accumulated sum reaches 90% probability.
Advantages
- —
Steers output style without retraining
- —
Easy to run locally on consumer chips (llama.cpp)
Disadvantages
- —
High temperatures increase chance of logical errors and hallucination
🎤 Interview questions
Explain how temperature scaling modifies the Softmax entropy. What is the mathematical limit as Temperature approaches zero?