Elamur Research
☰
home / papers / 2007 / METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W07-0734.pdf

Summary

The paper tackles a known weakness of automatic MT evaluation: Bleu gives unreliable scores for individual sentences. Meteor addresses this by aligning a system translation to a reference word by word. Matching runs in stages: identical words first, then words that share a Porter stem, then WordNet synonyms. From the alignment, Meteor computes a recall-weighted harmonic mean of unigram precision and recall. It then lowers that score with a penalty for fragmented word order. Three parameters (α, β, γ) control the metric. They had been fixed in 2004, and this release re-tunes them with a hill-climbing search that maximizes Pearson correlation with human adequacy and fluency judgments, using leave-one-system-out training. The release also adds Spanish, French and German versions. These rely on Porter stemmers only, because no free WordNets were available for those languages. On English test data, the tuned settings give modest but statistically significant gains in segment-level correlation. For the other languages, the optimal parameters differ clearly from the English ones. Together, these results suggest that evaluation metrics benefit from tuning to the target language and to the quality criterion being measured.

Key points

  • Score formula: score = (1 − Pen) · Fmean, where Pen = γ · (chunks/matches)^β. The previous defaults were α = 0.9, β = 3.0 and γ = 0.5.
  • English data: training used NIST 2003 Arabic-to-English (3,978 judgments, 6 systems) and testing used NIST 2004 Arabic-to-English (347 judgments, 5 systems). Spanish, French and German used WMT-06 English-to-X data (729 to 1,201 judgments).
  • Tuned English parameters (α/β/γ): adequacy 0.82/1.0/0.21, fluency 0.78/0.75/0.38, and adequacy+fluency sum 0.81/0.83/0.28. Fluency tuning places more weight on precision and on the word-order penalty, though recall still dominates.
  • English test correlations: adequacy rose from 0.6123 to 0.6191, fluency from 0.4355 to 0.4502, and the sum from 0.5704 to 0.5818. Bootstrap tests show these differences are significant at the 95% level.
  • Other languages: the Spanish, French and German parameters were tuned only on small training sets and lack unseen test data, so their stability is still unverified. Meteor is distributed freely by CMU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

387 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 368 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lavie2007meteor,
  title = {{METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments}},
  author = {Lavie, Alon and Agarwal, Abhaya},
  booktitle = {Proceedings of the Second Workshop on Statistical Machine Translation},
  year = {2007},
  url = {https://aclanthology.org/W07-0734/}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. ME-TEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures Detailed tables are not included for lack of space. for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June. find
  2. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence Estimation for Machine Translation. Technical Report Natural Language Engineering Workshop Final Report, Johns Hopkins University. find
  3. Marvin Humphrey. 2007. Perl Interface to Snowball Stemmers. http://search.cpan.org/ creamyg/Lingua-Stem-Snowball-0.941/lib/Lingua/Stem/Snowball.pm. ↗
  4. Alon Lavie, Kenji Sagae, and Shyamsundar Jayaraman. 2004. The Significance of Recall in Automatic Metrics for MT Evaluation. In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (AMTA-2004), pages 134–143, Washington, DC, September. find
  5. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In Proceedings of the Thirteenth Conference of the European Chapter of the Association for Computational Linguistics. find
  6. I. Dan Melamed, Ryan Green, and Joseph Turian. 2003. Precision and Recall of Machine Translation. In Proceedings of the HLT-NAACL 2003 Conference: Short Papers, pages 61–63, Edmonton, Alberta. find
  7. George Miller and Christiane Fellbaum. 2007. Word-Net. http://wordnet.princeton.edu/. ↗
  8. Franz Josef Och. 2003. Minimum Error Rate Training for Statistical Machine Translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. find
  9. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311– 318, Philadelphia, PA, July. find our card
  10. Martin Porter. 2001. The Porter Stemming Algorithm. http://www.tartarus.org/ martin/PorterStemmer/index.html. ↗
  11. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA-2006), pages 223–231, Cambridge, MA, August. find our card
  12. C. van Rijsbergen, 1979. Information Retrieval. Butterworths, London, UK, 2nd edition. find