Elamur Research⌕
☰
home / concepts / BLEU score

🎯 BLEU score

Concept · 26 key papers · all concepts

BLEU score (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It rates a candidate text by how many of its word n-grams also occur in one or more human reference translations, and it penalises candidates that are too short.

Also: BLEU, Bleu, BLEU-4, IBM BLEU, Bilingual Evaluation Understudy

What it is

BLEU estimates translation quality without a human judge. It compares system output with reference translations and counts shared word sequences. The score runs from 0 to 1 and is often reported multiplied by 100. A higher score means more overlap with the references. BLEU is meant to be computed over a whole test corpus, not a single sentence. It was proposed as a cheap, repeatable stand-in for human evaluation of machine translation (Papineni et al. 2002).

The metric measures surface agreement only. It does not check whether meaning is preserved, whether grammar is correct, or whether a different but valid wording was used. Its value rests on the assumption that, averaged over many sentences, overlap with references tracks human preferences. Much later work has tested and questioned that assumption; the main findings are summarised under Open questions (Reiter 2018).

The key idea

BLEU combines three components: clipped ("modified") n-gram precision, a geometric mean across n-gram orders, and a brevity penalty.

BLEU = BP · exp( Σₙ wₙ · log pₙ ),   n = 1 … N

pₙ = Σ min(count_cand(g), max_ref count(g)) / Σ count_cand(g)

BP = 1             if c > r
BP = exp(1 − r/c)  if c ≤ r

In these formulas, g is an n-gram of length n in the candidate. count_cand(g) is how often g appears in the candidate. max_ref count(g) is the largest number of times g appears in any single reference. Taking the minimum of the two is called clipping. It stops a system from gaining credit by repeating a common word: "the the the the" cannot earn more matches for "the" than a reference actually contains.

The sums run over the corpus, so pₙ is the modified precision for order n. N is the highest order, usually 4, which gives the common variant BLEU-4. The weights wₙ are usually uniform (1/N), which turns the exponentiated weighted sum of logs into a geometric mean.

The brevity penalty BP depends on two lengths. c is the total candidate length. r is the effective reference length, usually the reference length closest to each candidate, summed over the corpus. Precision alone would reward very short outputs, so BP shrinks the score when the candidate is shorter than the references. For instance, a five-word candidate against a six-word reference gets BP = exp(1 − 6/5) ≈ 0.82.

A worked example uses the reference "the cat is on the mat" and the candidate "the cat sat on the mat".

OrderMatched / candidate n-gramspₙ
1the, the, cat, on, mat → 5/60.833
2the cat, on the, the mat → 3/50.600
3on the mat → 1/40.250
4none → 0/30

Both sentences have six words, so BP = 1. With N = 3, the product 0.833 × 0.6 × 0.25 is 0.125; its cube root is 0.5, so BLEU-3 = 0.5. With N = 4, a single zero precision makes the whole product zero.

This is why sentence-level BLEU needs smoothing and why the metric is defined at corpus level. Smoothing means adding small pseudo-counts so that one empty n-gram order does not force the score to zero.

History and founding papers

BLEU was introduced by Papineni, Roukos, Ward and Zhu at IBM in 2002 (Papineni et al. 2002). It became the dominant automatic measure in machine translation research. More than a decade later, automatic evaluation for most target languages still depended on it (Denkowski & Lavie 2014). Its n-gram counting idea soon spread to other tasks.

Lin and Hovy turned BLEU's clipped precision into a recall measure for summary evaluation. They found that unmodified BLEU correlated with human rankings on single-document summaries from DUC 2001 (the Document Understanding Conference evaluation). The Spearman ρ was 0.66 using one reference and 0.82 using three (Lin & Hovy 2003). That line of work led to the ROUGE package (Lin 2004).

Critiques of BLEU appeared early and set the agenda for successor metrics:

How it is used today

BLEU remains a standard reported number in translation papers and shared tasks. It is also used as an optimisation target. Phrase-based systems commonly tuned feature weights with minimum error rate training (MERT), which searches for the weights that maximise an automatic score such as BLEU on development data. In one such system, the tuning setup made the output too short. This badly hurt scores under NIST, a separate n-gram metric that weights matches by how informative they are (Koehn et al. 2005).

Shared tasks paired BLEU with human ratings. In the 2006 European-language task, BLEU was computed against a single reference. Averaged per language pair, it ran from 28.33 for English–French, the easiest pair, to 14.01 for English–German, the hardest. The same task found that BLEU undervalued a rule-based commercial system on in-domain text (Koehn & Monz 2006).

Because tokenisation and casing change the score, organisers often normalise outputs themselves (Koehn & Monz 2006). Post called for clearer reporting of exactly how BLEU numbers are produced (Post 2018).

Several later metrics now appear next to BLEU, and often in place of it:

Outside translation, BLEU-4 is still a common baseline in image captioning. Reference-free scoring with CLIPScore complements it rather than replacing it: an information-gain analysis shows CLIPScore adding information beyond reference-based metrics such as BLEU-4. On New York Times news captions, reference-based metrics do far better: BLEU-4 reached 91% accuracy where CLIPScore reached 65% (Hessel et al. 2021).

Its use elsewhere has been strongly contested. For dialogue response generation, BLEU showed weak or no correlation with human judgments (Liu et al. 2016). In natural language generation, word-overlap metrics including BLEU correlated only weakly with human ratings (Novikova et al. 2017).

Open questions

The main open issue is validity: how far BLEU reflects what people value. A structured review concluded that the evidence supports BLEU for diagnostic, system-level evaluation of MT systems. It does not support BLEU for judging individual texts, for other NLG tasks, or for testing scientific hypotheses (Reiter 2018). Mathur, Baldwin and Cohn showed that meta-evaluation can itself mislead. A few outlier systems can inflate Pearson correlations, and small BLEU differences between systems often disagree with human judgments (Mathur et al. 2020).

Specific blind spots are well documented:

A further question is whether a metric that ranks outputs well also makes a good training objective. The evidence from Meteor tuning suggests the two goals can diverge (Denkowski & Lavie 2011).

As learned metrics take over, BLEU's remaining advantages are transparency, speed and comparability with decades of earlier results. How to weigh those against weaker agreement with human judgment is still debated.

Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

More papers

Topics: 🔁 Machine translation · 📏 Evaluation & benchmarks

Shelves: 🌐 Machine translation · 💬 Natural language processing

This page in Markdown → https://md.research.elamur.ai/free/en/concepts/bleu/