intermediate~4h
LM Evaluation Metrics: Entropy, Cross-Entropy & Perplexity
The information-theoretic foundation of language model evaluation: entropy, cross-entropy, perplexity, and bits-per-character — and why these metrics are both powerful and limited.
4
Subtopics
1
Exercises
1
Projects
5
Quiz Qs
4
Flashcards
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Derive perplexity from cross-entropy and explain its meaning as a branching factor
- •Compute sliding-window perplexity for documents longer than the model context length
- •Identify when BPC normalization is required vs. raw perplexity comparison
- •Explain why low perplexity does not guarantee good downstream task performance
- •Choose the right test set and tokenization strategy for a fair PPL comparison
LM Evaluation Metrics: Entropy, Cross-Entropy & Perplexity is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
📂 Subtopics
1234
Bits-per-Character: Normalizing Cross-Entropy Across Different Tokenizers
Using Perplexity to Compare Language Models in Practice: Held-Out Sets and Domain Sensitivity
Why These Metrics Are Both Powerful and Limited: What Perplexity Doesn't Measure
Calibration Metrics: Expected Calibration Error and Brier Score
Related concepts
cross-entropy lossNLL training objectiveKL divergencetokenizationbits-per-characterShannon entropyperplexity benchmarkssliding-window evaluationdata contaminationmodel evaluation
Next to learn
ai-as-judge →evaluation-pipeline-design →model-selection-benchmarking →