METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

What it is about

The paper proposes METEOR, a sentence-level metric built on explicit word-to-word alignment between the system output and each reference. The alignment is made in stages: exact surface match first, then stem match, then synonym match, each stage using only words not yet aligned, with ties broken by the fewest crossing links. From the alignment the metric takes unigram precision and recall, combines them into a recall-weighted harmonic mean, and multiplies by a penalty for fragmentation: matched words are grouped into chunks, and the penalty rises with the number of chunks, up to half the score. The authors argue that BLEU's geometric mean fails at sentence level and that its brevity penalty does not compensate for lost recall. On the LDC TIDES 2003 Arabic and Chinese data with human adequacy and fluency scores, segment-level Pearson correlation was about 0.35 for Arabic and 0.33 for Chinese, higher than unigram precision, recall or their F1, and recall alone was already better than BLEU at system level.

Ideas

I1. Make the alignment staged in tools/translate/src/measure.rs: exact match, then stem, then synonym from a BTreeMap concept table. The translator's meta-world already merges spellings and synonyms (thou, you, ye), so its concept table is the natural synonym source. Count matches per stage, so each report says where a match came from.
I2. Use the fragmentation penalty as an order measure. Count the chunks of matched words in the output; a linearizer change in en/src/lin.rs that increases the chunk count will show in this number without touching the exact-match score.
I3. Compute the recall-weighted form as a dev check. The paper finds recall the most important part, which bears on the translator's open question of dropped function words (problem 4).
I4. Count crossing alignments, the paper's measure of reordering, as a probe for the Hebrew verb-subject order (open problem 7).

For Mova

The code is tools/translate/src/measure.rs. The staged alignment is the cleanest way to bring the meta-world's concepts into a scorer without changing exact mode, and the stage counts make the scorer's output inspectable, which matches the translator's goal of saying why. Run it on the dev split twice before any claim, as the metric has tunable parameters that the paper itself did not train.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W05-0909.pdf

Summary

The authors present METEOR, an automatic metric for evaluating machine translation (MT). It is designed to address the weaknesses of BLEU: it ignores recall, accounts for word order only indirectly through higher-order n-grams, lacks explicit word-to-word matching, and assigns zero scores to individual sentences because of geometric averaging. METEOR builds a word alignment between the system translation and the reference in stages. First, identical words are matched, then words sharing a stem according to the Porter stemmer, and then synonyms from WordNet. If several alignments are equally large, the one with the fewest crossings is chosen. The score combines unigram precision, unigram recall and a fragmentation penalty, that is, a penalty for how scattered the matched words are. If there are multiple references, the best result is taken. The metric was tested on LDC TIDES 2003 data (Arabic and Chinese → English) by computing the Pearson correlation with human judgments at the level of individual segments. METEOR substantially outperformed precision and recall alone, as well as their combination F1, and each matching module improved the result. The work is important because it focuses on the sentence level: this is precisely where a metric must detect small differences between systems or between versions of the same system.

Key points

Pearson correlation with human judgments at the segment level: 0.347 for the Arabic and 0.331 for the Chinese LDC TIDES 2003 dataset.
Module ablation on the Arabic data: exact alone gives 0.312, with Porter 0.329, with WN-Stem 0.330, and the full sequence exact + Porter + WN-Synonymy 0.347. On the Chinese data, the same configurations give 0.293, 0.318, 0.312 and 0.331.
Human judgments of individual sentences are very noisy. When they were normalized by a group from MITRE Corporation, the correlation rose to 0.403 (Arabic) and 0.365 (Chinese).
In previous work, the authors showed that recall matters more than precision for agreement with humans. METEOR therefore accounts for recall directly rather than replacing it with a brevity penalty, as BLEU does.
The authors' plans for future work: tuning the penalty and scoring formulas on training data, matching words by semantic similarity, assigning different weights to matches from different modules, and using synthetic reference translations.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{banerjee2005meteor,
  title = {{METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments}},
  author = {Banerjee, Satanjeev and Lavie, Alon},
  booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
  year = {2005},
  url = {https://aclanthology.org/W05-0909/}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

George Doddington. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-occurrence Statistics. In Proceedings of 2 nd Human Language Technologies Conference (HLT-02). San Diego, CA. pp. 128-132. find
Margaret King, Andrei Popescu-Belis and Eduard Hovy. 2003. FEMTI: Creating and Using a Framework for MT Evaluation. In Proceedings of MT Summit IX, New Orleans, LA. find
Alon Lavie, Kenji Sagae and Shyamsundar Jayaraman, 2004. The Significance of Recall in Automatic Metrics for MT Evaluation. In Proceedings of AMTA-2004, Washington DC. September 2004. find
Bo Pang, Kevin Knight and Daniel Marcu. 2003. Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences. In Proceedings of HLT-NAACL 2003. Edmonton, Canada. May 2003. find our card
Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40 th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, PA. July 2002. pp. 311-318. find our card
Joseph P. Turian, Luke Shen and I. Dan Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX, New Orleans, LA. find
C. van Rijsbergen. 1979. Information Retrieval. Butterworths. London, England. 2 nd Edition. find