LM Evaluation Metrics: Entropy, Cross-Entropy & Perplexity
The information-theoretic foundation of language model evaluation: entropy, cross-entropy, perplexity, and bits-per-character — and why these metrics are both powerful and limited.
G-Eval Rubric Weights:
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Derive perplexity from cross-entropy and explain its meaning as a branching factor
- •Compute sliding-window perplexity for documents longer than the model context length
- •Identify when BPC normalization is required vs. raw perplexity comparison
- •Explain why low perplexity does not guarantee good downstream task performance
- •Choose the right test set and tokenization strategy for a fair PPL comparison
What is it?
Evaluation metrics for language models aren't arbitrary scores — they're grounded in information theory. The three foundational metrics are: entropy (the minimum bits a perfect compressor needs per token), cross-entropy (how many bits our model needs when the true data distribution generates text), and perplexity (the exponentiated cross-entropy — an interpretable 'average branching factor'). Bits-per-character (BPC) normalizes perplexity to the character level so you can fairly compare models with different vocabularies and tokenizers. These metrics come directly from Shannon's information theory and are identical to the NLL (negative log-likelihood) training loss averaged over a held-out test set.
Why it exists
Training a language model produces a probability distribution over vocabulary at each token position. To compare two models or track training progress, you need a single scalar that captures how well the model's distribution approximates the true data distribution. Accuracy (1 token right/wrong) is too coarse. Raw loss values depend on batch size and dataset scale, making them hard to compare across runs. Cross-entropy and perplexity fill this gap: they measure the model's average surprise when it sees ground-truth text, giving a continuous, differentiable signal that correlates with generation quality. Because perplexity directly measures compression efficiency, it has a clean information-theoretic interpretation that accuracy-based metrics lack.
Problem it solves
- How do we compare two language models without running expensive human evaluations?
- How do we track training progress when raw loss is scale-dependent?
- How do we compare models with different vocabularies or tokenization schemes?
- How do we set early stopping criteria during fine-tuning?
- How do we detect domain shift — when a model is likely to underperform on a new corpus?
Intuition
Imagine guessing the next word in a sentence. If you're good at it, you're rarely surprised — you assign high probability to the correct word. Perplexity measures your average surprise level over a long text.
A perplexity of 10 means: on average, you were choosing among 10 equally likely words at each position. A perplexity of 3 means you were very confident — only 3 plausible continuations. A perplexity of 1 is a perfect oracle model. A perplexity equal to vocabulary size (50,000+) means pure random guessing.
Key insight from Chip Huyen (AI Engineering, Ch.3): a model that achieves low perplexity on a test set has implicitly learned the statistical regularities of that text. But perplexity is a proxy — it measures compression quality, not task performance. A model can achieve PPL=5 on Wikipedia and still be useless at following instructions.
Analogy
Think of cross-entropy as a compression test.
You built a ZIP compression algorithm (your language model). Someone gives you a text to compress that you've never seen before (the test set). Cross-entropy (in bits/token) measures how many bits per token your compressor needs on this new text.
A perfect compressor (one that knows the true distribution P) needs exactly entropy(P) bits/token. If your compressor's model Q differs from P, it needs more bits — that extra cost is the cross-entropy gap H(P,Q) - H(P) = KL divergence.
Perplexity converts this compression cost into a branching factor: 'on average, how many words did your model think were equally likely at each position?' PPL=10 means your compressor was as uncertain as if it had to choose uniformly from 10 words each time.
Technical explanation
Shannon entropy H(P) = -Σₓ P(x) log₂ P(x) is the expected minimum bits to encode a sample from P. For natural English at character level: ~1.3 bits/char.
Cross-entropy H(P, Q) = -Σₓ P(x) log₂ Q(x) is the expected bits needed when the true distribution is P but we encode with a codebook optimized for Q. Since we never have P analytically, we estimate with the empirical test distribution: H ≈ -(1/N) Σᵢ log Q(xᵢ) where xᵢ are ground-truth tokens from the held-out test set. This is exactly the NLL training loss averaged over tokens — cross-entropy loss and NLL are the same thing.
KL divergence D_KL(P || Q) = H(P,Q) - H(P) ≥ 0 (Gibbs inequality). It is the 'wasted bits' from using the wrong distribution. D_KL = 0 iff Q = P everywhere.
Perplexity: PPL = exp(H) [natural log] or 2^H [log₂] The branching-factor interpretation: a PPL of k means the model was, on average, as uncertain as if it had to choose uniformly from k equally likely tokens.
Autoregressive LM perplexity over a sentence w₁...wᵀ: PPL = exp(-(1/T) Σᵢ log P(wᵢ | w₁...wᵢ₋₁))
Bits-per-character normalization: BPC = H_token / avg_chars_per_token Required when comparing models with different tokenizers. A model with a 100k-token vocabulary (GPT-4) sees fewer tokens per sentence than a 32k vocab model (LLaMA) but each token covers more characters — raw PPL is lower but not directly comparable.
Sliding-window perplexity: For documents longer than context length n, use stride s (typically n/2). The model processes a window of n tokens, but only the last s tokens count toward the NLL. The first n-s tokens serve as context warm-up. This avoids penalizing the model for missing tokens that fall outside its context window.
Benchmark references:
- GPT-2 (1.5B params): PPL ≈ 17.48 on Penn Treebank (Meister & Cotterell, 2021)
- GPT-3 (175B params): PPL ≈ 20.50 on Penn Treebank (different train/test split)
- LLaMA 2 (70B): PPL ≈ 5.83 on WikiText-2 (Meta AI, 2023)
- Note: these are not directly comparable due to different train/test splits and tokenizers
Architecture
Perplexity Evaluation Pipeline:
┌───────────────────────────────────────────────────────┐ │ TEST SET (fixed, held-out, tokenized) │ │ 'The cat sat on the mat' │ │ → tokens: [The=1, cat=2, sat=3, on=4, the=5, mat=6] │ └─────────────────────┬─────────────────────────────────┘ │ ▼ ┌───────────────────────────────────────────────────────┐ │ MODEL (autoregressive, causal attention) │ │ Input: [, The, cat, sat, on, the] │ │ Output: logits → softmax → P(next token) │ │ P(cat|The) = 0.10, P(sat|The,cat) = 0.30, ... │ └─────────────────────┬─────────────────────────────────┘ │ ▼ ┌───────────────────────────────────────────────────────┐ │ LOG PROBABILITY EXTRACTION │ │ log P(cat|The) = log(0.10) = -2.303 │ │ log P(sat|The,cat) = log(0.30) = -1.204 │ │ Sum: Σ log P(wᵢ|w₁...wᵢ₋₁) over all tokens │ └─────────────────────┬─────────────────────────────────┘ │ ▼ ┌───────────────────────────────────────────────────────┐ │ NLL → PERPLEXITY │ │ NLL = -(1/N) × Σ log P(wᵢ|w<ᵢ) │ │ PPL = exp(NLL) │ │ │ │ Good model (English Wikipedia): PPL 5-20 │ │ GPT-2 (1.5B) on Penn Treebank: PPL 17.48 │ │ LLaMA 2 (70B) on WikiText-2: PPL 5.83 │ └───────────────────────────────────────────────────────┘
Workflow
-
Choose a test set:
- MUST be held out (zero overlap with training data)
- Use the same tokenizer as the model you're evaluating
- Match the domain you care about (code PPL ≠ prose PPL)
- Common: WikiText-2, WikiText-103, Penn Treebank, The Pile
-
Tokenize test set and compute token IDs
-
Run forward passes (no gradient computation): for batch in test_loader: with torch.no_grad(): outputs = model(batch.input_ids, labels=batch.input_ids) nll_sum += outputs.loss * batch.num_tokens total_tokens += batch.num_tokens
-
Compute final PPL: ppl = torch.exp(nll_sum / total_tokens)
-
For long documents (> context_length tokens): Use sliding-window perplexity (stride = context_length // 2) to avoid edge effects from truncation
-
Normalize for cross-model comparison: If tokenizers differ, compute BPC = H / avg_chars_per_token
-
Report results:
- Specify test set name, tokenizer, and context length used
- Report on multiple domains if multi-domain performance matters
Example
import torch from transformers import AutoModelForCausalLM, AutoTokenizer def compute_perplexity(model_name: str, test_text: str, stride: int = 512) -> float: tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16) model.eval() encodings = tokenizer(test_text, return_tensors='pt') max_length = model.config.max_position_embeddings seq_len = encodings.input_ids.size(1) nlls, prev_end = [], 0 for begin_loc in range(0, seq_len, stride): end_loc = min(begin_loc + max_length, seq_len) trg_len = end_loc - prev_end # tokens actually scored input_ids = encodings.input_ids[:, begin_loc:end_loc] target_ids = input_ids.clone() target_ids[:, :-trg_len] = -100 # mask warm-up context with torch.no_grad(): outputs = model(input_ids, labels=target_ids) # outputs.loss is mean NLL over non-masked tokens nlls.append(outputs.loss * trg_len) prev_end = end_loc if end_loc == seq_len: break ppl = torch.exp(torch.stack(nlls).sum() / seq_len) return ppl.item() # BPC for cross-tokenizer comparison def perplexity_to_bpc(ppl: float, avg_chars_per_token: float) -> float: import math nll_nats = math.log(ppl) nll_bits = nll_nats / math.log(2) return nll_bits / avg_chars_per_token
Real-world usage
-
MLOps teams track validation PPL throughout training to detect overfitting: if validation PPL stops decreasing or increases while training PPL keeps dropping, the model is memorizing training data.
-
Model selection: compare two fine-tuned checkpoints on a domain-specific test set using PPL before running expensive human evaluations. PPL provides fast, cheap signal for pruning candidate models.
-
Chip Huyen (AI Engineering, Ch.3) notes that PPL is a proxy metric — good PPL doesn't guarantee good downstream task performance (instruction following, tool use, reasoning), but bad PPL is a reliable signal of a broken model.
-
LLaMA 2 paper (Meta AI, 2023): reported PPL on WikiText-2, Code, and multiple domains to establish baseline quality before instruction tuning. Used BPC normalization for cross-model comparisons.
-
Sliding-window PPL is standard for evaluating long-context models (32k-128k context) where naive truncation would mask the model's actual capability at long range.
Trade-offs
Stride length in sliding-window PPL: larger stride → less overlap → less computation, but more positional bias at the start of each window. Typical: stride = context_length // 2.
Context length vs. VRAM: longer context lowers PPL (model has more context to use) but requires proportionally more GPU memory for the key-value cache.
Test set domain choice: PPL on your specific domain (medical, legal, code) is far more predictive than general-corpus PPL. A model with PPL=8 on Wikipedia and PPL=40 on medical notes is the wrong choice for a healthcare application.
PPL vs. task metrics: PPL is cheap to compute continuously but correlates poorly with instruction-following benchmarks (MMLU, HumanEval). Use both: PPL for early training signal, task benchmarks before deployment.
Visual explanation
Information-Theoretic Hierarchy:
True distribution P(x): P(the)=0.30, P(a)=0.20, P(cat)=0.10, ... Model distribution Q(x): Q(the)=0.25, Q(a)=0.15, Q(cat)=0.20, ...
Entropy H(P) = -Σ P(x) log₂ P(x) [perfect compressor's cost] Cross-entropy H(P,Q) = -Σ P(x) log₂ Q(x) [our model's cost] KL divergence D_KL = H(P,Q) - H(P) [extra cost from mismatch]
In practice (we don't have P): H ≈ -(1/N) Σᵢ log Q(xᵢ) [average NLL over N test tokens]
Perplexity: PPL = exp(H) [natural log, used by PyTorch/HF] PPL = 2^H [log base 2, used in information theory papers]
Scale reference: H = 0.0 bits → PPL = 1 (perfect, omniscient model) H = 1.0 bits → PPL = 2 (knows it's 1 of 2 words) H = 3.0 bits → PPL = 8 (choosing among 8 words on average) H = 10.0 bits → PPL = 1024 (nearly random)
Bits-per-character: BPC = H_token × (tokens per character) [≈ H / avg_chars_per_token] English character-level entropy ≈ 1.3 bits/char (Shannon, 1951)
Sliding-window PPL for long docs (stride s, window n): [<────── context (warm-up) ──────>|<── scored ──>] t=0 t=n-s t=n-s+1 t=n Only tokens in the 'scored' window count toward NLL.
Advantages
- —
Directly linked to training objective (NLL loss) — no hidden mismatch between metric and what the model optimizes
- —
Fast to compute: a single forward pass over the test set, no human annotators required
- —
Single interpretable number with a clean meaning (average branching factor)
- —
Normalizable to BPC for fair cross-model, cross-tokenizer comparison
- —
Sensitive to training: PPL tracks learning curves reliably throughout training
- —
Universally reported in LLM papers — makes benchmark comparison possible
Disadvantages
- —
Dataset-dependent: PPL on Wikipedia ≠ PPL on medical texts — must always specify the test set
- —
Doesn't capture instruction-following ability, factuality, or safety
- —
Tokenizer-dependent: models with larger vocabularies have lower raw PPL by default
- —
Cannot catch memorization: if the model has seen the test set during training, PPL is artificially low
- —
PPL differences can be statistically insignificant — 5.0 vs 5.5 may be noise on a small test set
- —
Poor predictor of chat/instruction model quality: instruction-tuned models often have HIGHER PPL on raw Wikipedia than base models, despite being far better at the tasks users care about
Common mistakes
- —
Comparing PPL across models without normalizing for tokenization. GPT-4 (100k vocab) vs. LLaMA 2 (32k vocab): GPT-4's lower PPL partly reflects denser tokenization, not just better modeling. Always compute BPC when comparing across vocabulary sizes.
- —
Evaluating on the training set (data contamination). This gives artificially low PPL that doesn't predict held-out performance. Always use a strictly held-out test set with no overlap with training data.
- —
Reporting only average PPL across a full dataset. A model with average PPL=10 might have PPL=50 on rare genres and PPL=5 on the majority domain. The tail matters — report per-domain breakdown for production model selection.
- —
Conflating low PPL with good model quality for downstream tasks. An instruction-tuned model often has HIGHER PPL on raw Wikipedia than its base model counterpart, because RLHF/SFT shifts the distribution away from raw web text. PPL is a proxy, not the target.
- —
Ignoring sliding-window evaluation for long documents. Scoring a 100k-token document with a 4096-token context model by simple truncation gives meaningless results for the 96k tokens that fall outside the window. Always use stride-based sliding PPL.
🎤 Interview questions
Explain the relationship between cross-entropy, NLL loss, and perplexity. Why are they essentially the same thing?
When would you use BPC instead of raw perplexity? Walk through the calculation.
📂 Subtopics
Bits-per-Character: Normalizing Cross-Entropy Across Different Tokenizers
Comparing two models' perplexity directly is misleading if they use different tokenizers — bits-per-character (BPC) normalizes cross-entropy to a tokenizer-independent unit, making cross-model comparison fair.
~13 min
Using Perplexity to Compare Language Models in Practice: Held-Out Sets and Domain Sensitivity
Perplexity is only meaningful relative to a specific held-out test set — the same model can look great or terrible depending on how closely that test set matches what the model was actually trained on.
~12 min
Why These Metrics Are Both Powerful and Limited: What Perplexity Doesn't Measure
Low perplexity means a model predicts held-out text well — it says nothing directly about factual correctness, helpfulness, or safety, which is exactly why perplexity coexists with BLEU/ROUGE/LLM-judge rather than replacing them.
~12 min
Calibration Metrics: Expected Calibration Error and Brier Score
A model can be accurate but overconfident, or humble but underconfident — calibration metrics (ECE, Brier score) measure whether a model's STATED confidence actually matches its real-world accuracy.
~13 min