Elamur Research
☰
home / papers / 2017 / Results of the WMT17 Metrics Shared Task

Results of the WMT17 Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-4755.pdf

Summary

The WMT17 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system output against reference translations, can replace human quality judgements. The organizers distributed outputs of systems from the WMT17 News Translation Task, the Neural MT training task, and the health-domain HimL project, covering 16 language pairs. Participants returned two kinds of scores: one for a whole test set (system level) and one for each sentence (segment level). These scores were correlated with two human gold standards. The first was Direct Assessment (DA), which now replaces relative ranking as the official WMT method. The second was the semantics-oriented HUME score. Pearson correlation was the main statistic, and Kendall's τ was used where sentence-level DA data were too sparse. The strongest metrics exceeded 0.95 average Pearson correlation at system level, while sentence-level agreement typically fell between 0.4 and 0.6. Metrics that use character-level information again performed well, and both trained and untrained approaches reached the top. The findings help researchers choose reliable, publicly available evaluation tools.

Key points

  • Seven baselines (NIST, BLEU, SentBLEU, CDER, TER, PER and WER) served as comparison points for the 14 metrics submitted by 8 research groups.
  • The data comprised newstest2017 and himltest2017:
    • newstest2017 has about 3,000 sentences per direction (2,001 for Chinese and Latvian).
    • himltest2017 has about 300 sentences each for English→Czech/German/Polish/Romanian.
    • Together they yielded 166 real system outputs, plus 10K hybrid systems per language pair.
  • Top performers included chrF, CharacTer, BEER, BLEND and the new MEANT 2.0. MEANT 2.0 also led on the small HUME segment-level evaluation.
  • On self-reported speed, most good metrics were fast. Only MEANT 2.0 was notably slower, needing up to about one second per sentence.
  • A data leak affected English→Polish HUME: 324 of 340 sentences in one system's file had appeared in the training data. Among the trained metrics, only AutoDA was affected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4755

@inproceedings{bojar2017results,
  title = {{Results of the WMT17 Metrics Shared Task}},
  author = {Bojar, Ondřej and Graham, Yvette and Kamran, Amir},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4755},
  url = {https://doi.org/10.18653/v1/w17-4755}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
  2. Melania Duma and Wolfgang Menzel. 2017. Uhh submission to the wmt17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. ↗ our card
  4. Yvette Graham and Timothy Baldwin. 2014. Testing for Significance of Increased Correlation with Human Judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar, October. Association for Computational Linguistics. find
  5. Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-Based Evaluation of Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1264– 1274, Austin, Texas, November. Association for Computational Linguistics. find our card
  6. Yvette Graham and Qun Liu. 2016. Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA. Association for Computational Linguistics. find
  7. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A Grain of Salt for the WMT Manual Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  8. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  9. Ondřej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016a. Ten Years of WMT Evaluation Campaigns: Lessons Learnt. In Proceedings of the LREC 2016 Workshop Translation Evaluation From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 27–34, Portorose, Slovenia, 5. find
  10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Ondřej Bojar, Yvette Graham, , and Amir Kamran Miloš Stanojević. 2016b. Results of the WMT16 Metrics Shared Task . In Proceedings of the First Conference on Machine Translation, Gothenburg, Sweden, April. Association for Computational Linguistics. find our card
  11. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  12. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate Evaluation of Segment-level Machine Translation Metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, Denver, Colorado. find
  13. David Mareček, Ondřej Bojar, Ondřej Hübsch, Rudolf Rosa, and Dusan Varis. 2017. Cuni experiments for wmt17 metrics task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  14. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28, 1. find
  15. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Haji, Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 1659–1666, Paris, France. European Language Resources Association. find our card
  16. Jan Hajič. 2004. Complex Corpus Annotation: The Prague Dependency Treebank. In Insight into Slovak and Czech Corpus Linguistics, Bratislava, Slovakia. Jazykovedný ústav Ľ. Štúra, SAV. find
  17. Jindřich Helcl and Jindřich Libovický. 2017. Neural Monkey: An open-source tool for sequence learning. The Prague Bulletin of Mathematical Linguistics, 107:5–17. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. find our card
  19. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation Between European Languages. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’06, pages 102–121, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  20. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find our card
  21. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In In Proceedings of EACL, pages 241– 248. find
  22. Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  23. Chi-kiu Lo, Philipp Dowling, and Dekai Wu. 2015. Improving evaluation and optimization of MT systems against MEANT. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find
  24. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  25. Chi-kiu Lo. 2017. Meant 2.0: Accurate semantic mt evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  26. Miloš Stanojević and Khalil Sima’an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find
  27. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment casict-dcu submission to wmt17 metrics task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  28. Andre Tättar and Mark Fishel. 2017. bleu2vec: the painfully familiar metric on continuous vector space steroids. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  29. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD, USA. Association for Computational Linguistics. find our card
  30. Yolanda Vazquez-Alvarez and Mark Huckvale. 2002. The reliability of the ITU-t p.85 standard for the evaluation of text-to-speech systems. In Proc. of IC-SLP - INTERSPEECH. find
  31. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation, Berlin, Germany, August. Association for Computational Linguistics. find our card
  32. Evan James Williams. 1959. Regression analysis, volume 14. Wiley New York. find
  33. Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015. CASICT-DCU Participation in WMT2015 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find