Elamur Research
☰
home / papers / 2014 / Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W14-3348.pdf

Summary

For most of the world's languages, automatic MT evaluation still relies on BLEU. Richer metrics such as Meteor need language-specific tools, like stemmers and WordNet, and they need human quality judgments to tune their weights. Few target languages have either. Meteor Universal removes both requirements. It builds two resources from the same parallel corpus used to train a phrase-based system. The first is a function word list made of very frequent target-side words. The second is a paraphrase table, created by pivoting through source-language phrases and then filtered with language-independent pruning rules. Instead of tuning weights for each language, it uses one parameter set fitted to pooled WMT12 human rankings across eight translation directions. This shared set favors recall, word choice over word order, content words, and exact matches, while still giving credit to paraphrases. On held-out targets, Russian (WMT13) and Hindi (WMT14), it correlates with human rankings better than BLEU. On seen languages it trails fully tuned Meteor only slightly. As a result, low-resource target languages can get stronger evaluation.

Key points

  • Resources come only from the training bitext and a Moses-format phrase table. A word counts as a function word if its relative frequency exceeds 10⁻³. Paraphrases use the pivot method of Bannard and Callison-Burch (2005). Paraphrase instances below probability 0.001 are pruned, and final paraphrases below 0.01 are also pruned.
  • The universal parameters were tuned to maximize Kendall's τ over 101,099 binary WMT12 ranking judgments. These cover cs-en, de-en, es-en, fr-en and the four reverse directions.
  • Results for WMT13 Russian (sentence-level τ): 0.128 for Meteor Universal versus 0.068 for BLEU, nearly double.
  • Results for WMT14 English–Hindi: 0.264 versus 0.227. On English, Meteor Universal scored 0.206, against 0.214 for language-specific Meteor and 0.124 for BLEU.
  • The tool ships in Meteor 1.5 under the GNU LGPL. The script new_language.py generates resources for a new language, which are then used with the -new option.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

440 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 415 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3348

@inproceedings{denkowski2014meteor,
  title = {{Meteor Universal: Language Specific Translation Evaluation for Any Target Language}},
  author = {Denkowski, Michael and Lavie, Alon},
  booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},
  year = {2014},
  doi = {10.3115/v1/w14-3348},
  url = {https://doi.org/10.3115/v1/w14-3348}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 597–604, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
  2. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada, June. Association for Computational Linguistics. find our card
  4. Boxing Chen, Roland Kuhn, and George Foster. 2012. Improving amber, an mt evaluation metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 59–63, Montréal, Canada, June. Association for Computational Linguistics. find
  5. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011. Tesla at wmt 2011: Translation evaluation and tunable metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 78– 84, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  6. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  7. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. of NAACL/HLT 2003. find
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic, June. Association for Computational Linguistics. find
  9. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  10. George Miller and Christiane Fellbaum. 2007. Word-Net. http://wordnet.princeton.edu/. ↗
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. find
  12. Martin Porter. 2001. Snowball: A language for stemming algorithms. http://snowball.tartarus.org/texts/. ↗
  13. Matthew Snover, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or HTER? Exploring different human judgments with a tunable MT metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 259–268, Athens, Greece, March. Association for Computational Linguistics. find
  14. C. J. van Rijsbergen, 1979. Information Retrieval, chapter 7. Butterworths, London, UK, 2nd edition. find
  15. Mengqiu Wang and Christopher Manning. 2012. Spede: Probabilistic edit distance metrics for mt evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 76– 83, Montréal, Canada, June. Association for Computational Linguistics. find