Temperature, Top-k & Top-p: What They Control and How to Tune Them

These three parameters all control randomness in generation, but they control it in genuinely different ways — mixing them up is one of the most common tuning mistakes.

Temperature works by reshaping the softmax function that turns the model's raw scores into a probability distribution, before any token gets picked. Think of it as a dial on how 'confident' the model's distribution looks. At low temperature (close to 0), probabilities concentrate hard around whatever token was already most likely — this produces nearly greedy, deterministic generation. At higher temperature (0.7-1.0), the reshaped distribution becomes flatter and more uniform, so previously-unlikely tokens get a real shot at being sampled — producing more creative, diverse, but also noisier output. As a rule of thumb: lower temperature suits QA and chatbot-style tasks where you want reliable, focused answers; higher temperature suits brainstorming and creative writing where variety is the point.

Top-k works differently — instead of reshaping the whole distribution, it restricts the CANDIDATE POOL before sampling even happens. The default behavior is to sample from all tokens, weighted by their probability. Top-k truncates this to only the k most probable tokens: with k=5, the model only ever considers its 5 most likely next tokens, no matter how many thousands of tokens exist in the vocabulary. This enforces focus, but pick k too small and you risk repetitive, low-diversity outputs because there's simply nowhere else for the model to go.

Top-p (nucleus sampling) restricts the candidate pool differently: instead of a fixed COUNT of tokens, it takes the smallest set of tokens whose cumulative probability reaches p. With top_p=0.9, if the model is very confident (one token has 80% probability), the pool might only need 2-3 tokens to reach 90% mass; if the model is uncertain (probability spread thin across many tokens), the pool could include dozens of tokens. This makes top-p more adaptive than a fixed top-k — it naturally widens the pool exactly when the model is uncertain and narrows it when the model is confident, which is often a better balance of coherence and diversity than a fixed k.

import numpy as np

def softmax_with_temperature(logits: np.ndarray, temperature: float) -> np.ndarray:
    scaled = logits / max(temperature, 1e-8)  # low temp -> sharper, high temp -> flatter
    exp = np.exp(scaled - np.max(scaled))     # subtract max for numerical stability
    return exp / exp.sum()

def top_k_filter(probs: np.ndarray, k: int) -> np.ndarray:
    top_k_idx = np.argsort(probs)[-k:]
    filtered = np.zeros_like(probs)
    filtered[top_k_idx] = probs[top_k_idx]
    return filtered / filtered.sum()  # renormalize over the surviving tokens

def top_p_filter(probs: np.ndarray, p: float) -> np.ndarray:
    sorted_idx = np.argsort(probs)[::-1]
    cumulative = np.cumsum(probs[sorted_idx])
    cutoff = np.searchsorted(cumulative, p) + 1  # smallest set reaching mass p
    keep_idx = sorted_idx[:cutoff]
    filtered = np.zeros_like(probs)
    filtered[keep_idx] = probs[keep_idx]
    return filtered / filtered.sum()

logits = np.array([4.0, 3.5, 1.0, 0.5, 0.2])  # 5-token toy vocabulary
print("temp=0.2:", softmax_with_temperature(logits, 0.2).round(3))  # sharp, near-greedy
print("temp=1.5:", softmax_with_temperature(logits, 1.5).round(3))  # flatter, more uniform