Using Perplexity to Compare Language Models in Practice: Held-Out Sets and Domain Sensitivity

~12 min read

Perplexity is only meaningful relative to a specific held-out test set — the same model can look great or terrible depending on how closely that test set matches what the model was actually trained on.

Using Perplexity to Compare Language Models in Practice: Held-Out Sets and Domain Sensitivity is a Pro topic

Key points

•Perplexity must be measured on a HELD-OUT test set the model wasn't trained on — measuring it on training data tells you almost nothing (the model may have memorized it)
•The same model can show very different perplexity numbers on different held-out sets, depending on how closely each set's domain matches the model's training data
•A perplexity number reported without specifying its held-out test set is essentially uninterpretable for comparison purposes
•Valid model comparisons require measuring every model against the SAME held-out set — comparing across different test sets isn't a fair comparison
•This is exactly why standardized benchmarks (WikiText-103, the Pile) are useful: not because any single number is intrinsically meaningful, but because everyone reports against the same fixed set