Information Theory Basics: Entropy, Cross-Entropy and KL Divergence

Here's a question that sounds abstract but has a very concrete answer: how many yes/no questions do you need, on average, to guess an outcome? If a coin always lands heads, you need zero questions — you already know the answer. If it's a fair coin, you need exactly one question. Entropy is the formal version of this idea: it measures the average 'surprise' (uncertainty) in a probability distribution, measured in bits. A distribution with entropy 0 is completely predictable (certain); a distribution spread evenly across many outcomes has HIGH entropy (very uncertain, very surprising on average). For a distribution p, entropy is H(p) = -sum(p(x) * log2 p(x)) over every outcome x.

Cross-entropy extends this to compare TWO distributions: the TRUE distribution p (what actually happens) and a PREDICTED distribution q (what your model thinks will happen). Cross-entropy H(p, q) measures the average surprise you'd experience using your model's predicted probabilities q to describe outcomes that are really drawn from p. If your model is a perfect predictor (q = p exactly), cross-entropy EQUALS plain entropy — the lowest it can possibly be. Any mismatch between q and p makes cross-entropy larger. This is precisely why cross-entropy loss is what every LLM is trained to minimize: at each step, the TRUE distribution is 'the actual next word was X' (100% probability on the real word), and the model's predicted distribution q is its softmax output over the vocabulary — training pushes q toward putting as much probability as possible on the real next word.

KL divergence (Kullback-Leibler divergence) measures the GAP between two distributions directly: KL(p || q) = H(p, q) - H(p), i.e. cross-entropy minus the true distribution's own entropy. It's always >= 0, and it's 0 only when p and q are identical. Where cross-entropy mixes in the 'baseline surprise' of p itself, KL divergence isolates JUST the extra surprise caused by q being wrong. This is why KL divergence shows up whenever you're comparing two distributions directly — for example, checking how much a fine-tuned model's output distribution has drifted from the original base model (used in RLHF to keep a model from straying too far during training).

import math

def entropy(p: dict) -> float:
    """Average surprise (bits) of distribution p."""
    return -sum(prob * math.log2(prob) for prob in p.values() if prob > 0)

def cross_entropy(p_true: dict, q_pred: dict) -> float:
    """Average surprise using q's probabilities to describe outcomes
    that really come from p. Minimized when q == p."""
    return -sum(p_true[x] * math.log2(q_pred[x]) for x in p_true if p_true[x] > 0)

def kl_divergence(p_true: dict, q_pred: dict) -> float:
    """The extra surprise caused specifically by q being wrong."""
    return cross_entropy(p_true, q_pred) - entropy(p_true)

fair_coin = {"heads": 0.5, "tails": 0.5}
biased_coin = {"heads": 0.9, "tails": 0.1}

print(f"Entropy of a fair coin:    {entropy(fair_coin):.3f} bits")   # 1.000 (max uncertainty)
print(f"Entropy of a biased coin:  {entropy(biased_coin):.3f} bits")  # ~0.469 (more predictable)

# LLM training scenario: true next word is 'cat' (100% probability),
# model predicts a softmax distribution over the vocabulary
true_word = {"cat": 1.0, "dog": 0.0, "fish": 0.0}
good_pred = {"cat": 0.85, "dog": 0.10, "fish": 0.05}
bad_pred  = {"cat": 0.10, "dog": 0.60, "fish": 0.30}

print(f"Cross-entropy loss (good model): {cross_entropy(true_word, good_pred):.3f}")
print(f"Cross-entropy loss (bad model):  {cross_entropy(true_word, bad_pred):.3f}")
# Lower cross-entropy = better predictions -- exactly what training minimizes