Automated Metrics: BLEU, ROUGE and BERTScore

The cheapest way to evaluate an LLM output is to compare it automatically against a known-good reference answer. These metrics are fast, deterministic, and free to run on thousands of examples — which is exactly why they became standard, and also why you have to understand their blind spots.

BLEU (Bilingual Evaluation Understudy) was built for machine translation. It measures precision of n-grams: of the word sequences the model produced, how many also appear in the reference? It counts matching unigrams, bigrams, trigrams, etc., and multiplies in a 'brevity penalty' so a model can't cheat by outputting one safe word. BLEU rewards outputs that use the same phrasing as the reference.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was built for summarization and flips the emphasis to recall: of the reference's content, how much did the model capture? ROUGE-N counts n-gram overlap; ROUGE-L uses the longest common subsequence to reward correct ordering without demanding exact adjacency. High ROUGE means the summary covered the reference's key words.

The shared weakness of both is that they operate on surface word-overlap. 'The film was excellent' and 'The movie was superb' mean the same thing but share almost no words, so BLEU and ROUGE score them poorly. Conversely, a fluent but wrong answer that happens to reuse the reference's vocabulary can score well. They reward lexical mimicry, not correctness or meaning.

BERTScore addresses the synonym problem by working in embedding space. It runs both the candidate and the reference through a pretrained model (like BERT), then matches each token to its most similar token in the other text by cosine similarity of their contextual embeddings. Because 'excellent' and 'superb' land near each other in embedding space, BERTScore recognizes them as a match. It correlates far better with human judgment than n-gram metrics — but it still needs a reference answer, is sensitive to which embedding model you pick, and can't judge factual correctness that the reference didn't spell out.

The honest summary: use these for cheap, high-volume regression checks and relative comparisons ('did this change make outputs closer to our references?'), not as a final verdict on quality. For open-ended generation where many different answers are all correct, reference-based metrics break down — which is exactly the gap the next subtopic (LLM-as-judge) fills.

# ROUGE-style n-gram overlap and a cosine-similarity stand-in for
# the idea behind BERTScore. (Real BERTScore uses contextual
# embeddings from a transformer; here we illustrate the concept.)

def ngram_overlap(candidate: str, reference: str, n: int = 1) -> float:
    """ROUGE-N-style recall: fraction of reference n-grams the candidate covers."""
    def ngrams(text):
        toks = text.lower().split()
        return [tuple(toks[i:i+n]) for i in range(len(toks) - n + 1)]
    cand, ref = ngrams(candidate), ngrams(reference)
    if not ref:
        return 0.0
    matches = sum(1 for g in ref if g in cand)
    return matches / len(ref)

cand = "The movie was superb"
ref  = "The film was excellent"
print(f"ROUGE-1 recall: {ngram_overlap(cand, ref):.2f}")  # low: few shared words

# BERTScore captures the synonymy that n-gram overlap misses:
#   from bert_score import score
#   P, R, F1 = score([cand], [ref], lang="en")
#   print(F1)  # high, because 'superb'~'excellent', 'movie'~'film' in embedding space