Why These Metrics Are Both Powerful and Limited: What Perplexity Doesn't Measure
~12 min read
Low perplexity means a model predicts held-out text well — it says nothing directly about factual correctness, helpfulness, or safety, which is exactly why perplexity coexists with BLEU/ROUGE/LLM-judge rather than replacing them.
Why These Metrics Are Both Powerful and Limited: What Perplexity Doesn't Measure is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Perplexity is cheap and powerful specifically because it needs no labels, references, or separate judge model — just held-out text and the model itself
- •Low perplexity measures good prediction of EXISTING text — it says nothing directly about factual correctness, helpfulness, or safety of GENERATED output
- •Instruction-tuned/RLHF-aligned models are no longer optimized purely for next-token cross-entropy, making generic-text perplexity an increasingly weak proxy for their real-world quality
- •This is exactly why perplexity coexists with BLEU/ROUGE/BERTScore, LLM-as-judge, and human evaluation rather than replacing them — each measures something genuinely different
- •A mature evaluation practice treats perplexity as one cheap intrinsic signal among several layers, never as the sole measure of whether a model is actually good