Information Theory Basics: Entropy, Cross-Entropy and KL Divergence
~14 min read
Entropy measures how 'surprising' a distribution is on average; cross-entropy measures how well a predicted distribution matches the true one — and it's the actual loss function used to train every LLM.
Information Theory Basics: Entropy, Cross-Entropy and KL Divergence is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Entropy measures the average 'surprise' or uncertainty in a distribution, in bits — 0 for a certain outcome, higher for more spread-out distributions
- •Cross-entropy H(p, q) measures the average surprise of using predicted distribution q to describe outcomes really drawn from true distribution p
- •Cross-entropy is minimized (equals plain entropy) exactly when q matches p perfectly — which is why it's the loss function every LLM trains to minimize
- •KL divergence = cross-entropy minus entropy: it isolates the EXTRA surprise caused specifically by q being wrong, and is always >= 0
- •KL divergence is used to measure how far a model's output distribution has drifted from a reference (e.g. keeping a fine-tuned model close to its base model in RLHF)