Probability in LLMs: Next-Token Prediction, Softmax and Perplexity
~13 min read
An LLM is, at its core, a probability machine: softmax turns raw scores into a valid next-token distribution, and perplexity turns that distribution's quality into one human-readable number.
Probability in LLMs: Next-Token Prediction, Softmax and Perplexity is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •At every generation step, an LLM computes a full probability distribution over its entire vocabulary — every idea in this unit converges here
- •Softmax turns raw, unbounded logits into a valid distribution: exponentiate (make positive) then normalize (sum to 1)
- •Sampling from that distribution (greedy = highest probability, or temperature-controlled sampling) picks the next token; a sentence's probability is the product of each token's conditional probability
- •Perplexity = 2^(cross-entropy loss) — turns the training loss into an intuitive 'how confused is the model' number
- •Perplexity of 1 = perfect prediction; perplexity near vocabulary size = random guessing; lower perplexity = better language-model fit