Conditional Probability & Why Temperature Exists

Before diving into how LLMs generate text, it helps to understand conditional probability: the probability of an event given that another event has occurred, written P(A|B) — 'probability of A given B.' A simple example: if we're predicting whether it will rain today (event A), knowing it's cloudy (event B) changes our prediction. Since rain is more likely when it's cloudy, P(A|B) is high.

This is exactly the question an LLM answers at every generation step. An LLM like GPT-4 is tasked with predicting the next word in a sequence — this is a question of conditional probability: given the words that have come before, what is the most likely next word? To predict the next word, the model calculates the conditional probability for each possible next word, given the previous words (the context), and the word with the highest conditional probability is a candidate for the prediction.

The LLM learns a high-dimensional probability distribution over sequences of words, and the parameters of this distribution are the trained weights. But there's a problem: if we always pick the word with the single highest probability, we end up with repetitive outputs, making the model far less useful and stifling its creativity. This is exactly where temperature comes in — instead of always selecting the single best token, the model 'samples' the prediction, so even if one token has the highest score, it may not be chosen. Temperature introduces a tweak to the softmax function that shapes how this sampling behaves: at low temperature, probabilities concentrate around the most likely token (nearly greedy generation); at high temperature, probabilities become more uniform, producing more random and stochastic outputs. This conditional-probability framing is the conceptual bridge between 'how LLMs fundamentally work' and the full sampling pipeline (temperature, top-k, top-p, min-p) covered in depth in this topic's main content.

# Conditional probability, made concrete for next-token prediction
# P(next_token | previous_tokens) — the model outputs one such distribution
# per generation step.

import torch
import torch.nn.functional as F

def next_token_distribution(logits: torch.Tensor) -> torch.Tensor:
    # logits: (vocab_size,) — the model's raw scores for 'what comes next'
    # given everything generated so far (the conditioning context)
    return F.softmax(logits, dim=-1)  # P(next_token | previous_tokens)

# Always picking argmax(distribution) = greedy = repetitive, low-diversity output
# Sampling from the distribution (optionally reshaped by temperature) instead
# preserves the model's learned diversity, avoiding the repetition problem.