Text Generation Metrics: BLEU, ROUGE, METEOR
How do you score generated text against a reference? These metrics count word overlap, which is fast and cheap and systematically disagrees with human judgment.
On this page
Scoring a label is easy: it matches or it does not. Scoring generated text is not, because a good translation or summary can be phrased a hundred valid ways that share few exact words with any single reference. BLEU, ROUGE, and METEOR are the classic attempts to automate this by measuring word overlap against one or more human references. They are fast and cheap, and understanding why they disagree with humans is more useful than the formulas.
BLEU: precision of n-grams
BLEU (bilingual evaluation understudy) was built for machine translation. It asks: of the n-grams the model produced, how many appear in the reference?
It counts overlap at several n-gram sizes — unigrams (single words), bigrams (word pairs), up to 4-grams — and combines them. Unigram overlap checks that the right words are present; higher n-grams check that they are in sensible local order. A 4-gram match means four consecutive words line up with the reference, which is decent evidence of fluent phrasing.
Two patches keep it honest:
- Clipping. Producing “the the the the” cannot score credit for “the” more times than it appears in the reference, so a model cannot farm points by repeating a common word.
- Brevity penalty. BLEU is precision-based, so a one-word output (“the”) could be trivially precise. The brevity penalty docks scores for outputs shorter than the reference, forcing coverage.
BLEU is a corpus-level metric — it is reliable averaged over a whole test set and noisy on any single sentence.
ROUGE: recall for summarization
ROUGE (recall-oriented understudy for gisting evaluation) is BLEU’s mirror image, built for summarization. Where BLEU asks “how much of what I generated is in the reference” (precision), ROUGE mostly asks “how much of the reference did I capture” (recall) — because a summary’s job is coverage.
The common variants:
- ROUGE-N: n-gram recall (ROUGE-1 for unigrams, ROUGE-2 for bigrams).
- ROUGE-L: longest common subsequence — rewards words appearing in the same relative order without requiring them to be adjacent, which is more forgiving of insertions.
ROUGE is usually reported as an F-score that blends precision and recall, but its heart is recall: did the summary cover the reference’s content.
METEOR: overlap with a brain
METEOR was designed to fix BLEU’s most obvious blindness — it matches only exact words, so “quick” and “fast” score zero overlap. METEOR adds:
- Stemming, so “running” matches “run”.
- Synonym and paraphrase matching, so “quick” matches “fast”.
- An explicit word-order penalty for fragmented matches.
Because it understands that different words can carry the same meaning, METEOR correlates better with human judgment at the sentence level than BLEU. It is also more expensive and language-dependent, since it needs stemmers and synonym resources.
Why they diverge from human judgment
This is the part that matters, and the reason the metrics survive mainly as fast proxies rather than ground truth. All three fundamentally count surface overlap, and surface overlap is not meaning.
Valid paraphrases score low. “The cat sat on the mat” and “The feline rested on the rug” mean the same thing and share almost no words. BLEU and ROUGE punish the second heavily. The metric mistakes a good answer for a bad one — the large-output-space problem from why evaluation is hard made concrete.
Fluent nonsense can score high. An output can overlap the reference’s words while being ungrammatical or factually wrong. High word overlap is not correctness, and these metrics cannot detect a hallucinated fact that happens to reuse reference vocabulary.
One reference is too few. With a single reference, every legitimate alternative phrasing is penalized. Multiple references help and are expensive to collect, so most evaluations use one and quietly accept the bias.
They are blind to meaning, faithfulness, and coherence — exactly the qualities that matter most for modern generation. A summary can be high-ROUGE and unfaithful to its source.
Where they still earn their place
They are not useless — they are the right tool for a narrow job. Use them when you need a fast, cheap, reproducible signal to compare system versions during iteration, especially in translation and summarization where references are natural. They track large regressions well and cost nothing to run on every commit.
For final quality judgments, pair or replace them with:
- Embedding-based metrics (e.g. BERTScore) that compare meaning via embeddings instead of exact words, catching paraphrases the n-gram metrics miss.
- Model-as-judge, which scales subjective scoring — with the caveats covered in evals basics: validate the judge, prefer pairwise comparison.
- Human evaluation, still the ground truth for faithfulness and coherence.
The honest framing: BLEU and ROUGE measure “does this look like the reference”, not “is this good”. Keep them for cheap regression detection and never let a BLEU delta be the last word on quality.
What to remember
- BLEU, ROUGE, and METEOR score generated text by word overlap against human references — fast, cheap, reproducible.
- BLEU is n-gram precision (translation), with clipping and a brevity penalty; ROUGE is recall-oriented (summarization); METEOR adds stemming and synonyms to reduce exact-match blindness.
- All count surface overlap, so they penalize valid paraphrases, reward fluent-but-wrong text, and ignore faithfulness — which is why they diverge from human judgment.
- One reference biases the score; more references help but cost more.
- Use them for regression detection during iteration; use embedding metrics, model-as-judge, or humans for final quality.