Why These Metrics Are Both Powerful and Limited: What Perplexity Doesn't Measure

The previous two subtopics covered how to compute and properly compare perplexity/cross-entropy; this subtopic covers the honest limits of what these information-theoretic metrics actually tell you — directly addressing this topic's own framing (per its title) of these metrics being 'both powerful and limited.'

What perplexity IS powerful at: it's a purely intrinsic measure, needing no human labels, no reference answers, and no separate evaluation model — just a held-out text sample and the model itself. This makes it extremely cheap to compute at scale and genuinely useful for tracking whether basic language-modeling ability is improving or regressing across training runs or model versions, exactly the way a fever thermometer is cheap and useful for tracking whether something in the body has changed, even though it can't diagnose WHAT changed.

What perplexity does NOT measure, and this is the critical limitation: a model can achieve genuinely excellent (low) perplexity on held-out text while still being factually wrong, unhelpful, or unsafe in its actual generated outputs. Perplexity only measures how well the model predicts EXISTING, already-written text — it says nothing about the quality of text the model GENERATES on its own when given a prompt, since generation and next-token prediction on reference text are related but genuinely distinct capabilities. A model can be superbly confident and well-calibrated on predicting the next word of Wikipedia articles while still producing hallucinated facts, unhelpful responses, or biased content when actually generating freely. Additionally, modern instruction-tuned and RLHF-aligned models (covered in this curriculum's sft-vs-rft and reinforcement-learning material) are no longer optimized purely to minimize next-token cross-entropy on generic text at all — their training objective shifts toward human preference and task success, which means perplexity on generic held-out text becomes an increasingly weak proxy for what actually matters about a chat-tuned model's real-world quality.

This is precisely why perplexity coexists with, rather than replaces, the rest of the evaluation stack covered in this curriculum's llm-evaluation material: reference-based text-overlap metrics (BLEU, ROUGE, BERTScore) measure how close a GENERATED output is to a known-good answer; LLM-as-judge measures open-ended output quality against a rubric when no single reference exists; human evaluation measures what real people actually think. Each layer of this evaluation stack measures something genuinely different — perplexity measures raw language-modeling fit on existing text, the other layers measure properties of actual GENERATED output quality that perplexity is structurally blind to. A mature evaluation practice uses perplexity as one cheap, fast intrinsic signal among several, never as the sole measure of whether a model is actually good.

# Illustrating the DISCONNECT: a model can have excellent (low)
# perplexity on held-out text while still generating factually
# wrong or unhelpful output -- these are genuinely different things.

def evaluate_intrinsic_perplexity(model_name: str, held_out_perplexity: float) -> str:
    return f"{model_name}: perplexity {held_out_perplexity:.1f} (measures fit to EXISTING text)"

def evaluate_generated_output_quality(model_name: str, factual_accuracy: float,
                                      helpfulness_score: float) -> str:
    return (f"{model_name}: factual accuracy {factual_accuracy:.0%}, "
            f"helpfulness {helpfulness_score:.0%} (measures GENERATED output quality)")

models = [
    {"name": "Model A", "perplexity": 8.2, "factual_accuracy": 0.60, "helpfulness": 0.55},
    {"name": "Model B", "perplexity": 9.5, "factual_accuracy": 0.88, "helpfulness": 0.91},
]

for m in models:
    print(evaluate_intrinsic_perplexity(m["name"], m["perplexity"]))
    print(evaluate_generated_output_quality(m["name"], m["factual_accuracy"], m["helpfulness"]))
    print()

print("-> Model A has BETTER (lower) perplexity but WORSE real-world quality --")
print("   perplexity alone would have picked the wrong model\n")

EVALUATION_STACK = {
    "perplexity (intrinsic)": "fit to existing held-out text -- cheap, no labels needed",
    "BLEU/ROUGE/BERTScore": "closeness of GENERATED output to a known-good reference",
    "LLM-as-judge": "open-ended generated output quality against a rubric, no reference needed",
    "human evaluation": "ground truth for what real people actually think",
}
print("A mature evaluation stack layers ALL of these -- no single layer suffices:")
for layer, purpose in EVALUATION_STACK.items():
    print(f"  {layer}: {purpose}")