Using Perplexity to Compare Language Models in Practice: Held-Out Sets and Domain Sensitivity

The probability-basics material establishes WHAT perplexity is (2 to the power of cross-entropy loss); this subtopic covers the practical methodology of actually USING it to evaluate and compare models — a methodological layer the pure definition doesn't address on its own.

Perplexity is always measured against a SPECIFIC held-out test set — a collection of text the model was NOT trained on, used specifically to check how well the model's learned probability distribution predicts genuinely new text. This 'held-out' requirement is not optional: measuring perplexity on text the model WAS trained on tells you almost nothing useful, since a model can achieve artificially low perplexity on text it has directly memorized, the same way a student who memorized the answer key would ace a test drawn from that exact key. This mirrors the train/validation split concept from the neural-networks and deep-learning prerequisite material, applied specifically to language-modeling evaluation.

The subtler, easy-to-miss issue is domain sensitivity: a model's perplexity number is only meaningful relative to the SPECIFIC held-out set it was measured against, and two different held-out sets can give wildly different numbers for the exact same model. A model trained mostly on web text and code might show excellent (low) perplexity on a held-out set of web articles, but noticeably worse (higher) perplexity on a held-out set of specialized legal contracts or medical literature — not because the model got worse, but because that text's vocabulary and structure is genuinely less similar to what the model has seen. This means a perplexity number reported in isolation ('Model X achieves a perplexity of 12 on WikiText') is only useful for comparison purposes when you know it was measured on the SAME held-out set as whatever else you're comparing it to — comparing Model X's WikiText perplexity against Model Y's perplexity on a completely different test set tells you nothing reliable.

The practical methodology this implies: when evaluating or comparing language models via perplexity, always use the SAME held-out test set across every model being compared, choose a test set that's representative of the DOMAIN you actually care about (not just whatever public benchmark set is convenient, if your real use case is narrower or more specialized), and treat any perplexity number reported without a clearly specified held-out set as essentially uninterpretable. This same-test-set discipline is exactly what makes standard perplexity benchmarks (like WikiText-103 or the Pile) useful in practice — not because any one number is intrinsically meaningful, but because everyone reporting a number against that same fixed benchmark IS making a fair, apples-to-apples comparison.

# Illustrating domain sensitivity: the SAME toy model evaluated on
# two different held-out sets gives very different perplexity numbers.
import math

def perplexity(token_probs: list[float]) -> float:
    bits_per_token = sum(-math.log2(p) for p in token_probs) / len(token_probs)
    return 2 ** bits_per_token

# Toy stand-in: probabilities a model assigns to the ACTUAL next tokens
# in each held-out set (higher probability = the model 'saw this coming')
web_text_held_out_probs = [0.7, 0.65, 0.72, 0.68, 0.75]      # familiar domain
legal_text_held_out_probs = [0.25, 0.20, 0.30, 0.22, 0.18]   # unfamiliar domain

web_perplexity = perplexity(web_text_held_out_probs)
legal_perplexity = perplexity(legal_text_held_out_probs)

print(f"SAME model, held-out web text:   perplexity = {web_perplexity:.2f}")
print(f"SAME model, held-out legal text:  perplexity = {legal_perplexity:.2f}")
print("-> Same model, wildly different numbers -- purely because of")
print("   how well each held-out SET matches the model's training distribution\n")

def is_valid_comparison(model_a_test_set: str, model_b_test_set: str) -> bool:
    """The core methodological rule: a perplexity comparison is only
    valid if BOTH models were measured on the exact same held-out set."""
    return model_a_test_set == model_b_test_set

print("Comparing Model A (WikiText-103) vs Model B (WikiText-103):",
      is_valid_comparison("WikiText-103", "WikiText-103"))
print("Comparing Model A (WikiText-103) vs Model C (a private legal corpus):",
      is_valid_comparison("WikiText-103", "private_legal_corpus"))