Automated Metrics: BLEU, ROUGE and BERTScore
~13 min read
Reference-based metrics score an output by comparing it to a 'gold' answer. BLEU and ROUGE count overlapping words; BERTScore compares meaning via embeddings. All are cheap but blind to correctness the reference didn't anticipate.
Automated Metrics: BLEU, ROUGE and BERTScore is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •BLEU measures n-gram precision (built for translation); ROUGE measures n-gram recall + longest-common-subsequence (built for summarization)
- •Both judge surface word-overlap, so they penalize correct paraphrases ('superb' vs 'excellent') and can reward fluent-but-wrong answers
- •BERTScore matches tokens by contextual-embedding cosine similarity, so it recognizes synonyms and correlates better with human judgment
- •All three are reference-based: they need a gold answer and can't judge correctness the reference didn't anticipate
- •Use them for cheap high-volume regression checks and relative comparisons, not as a final quality verdict on open-ended generation