Elamur Research
☰
home / papers / 2018 / Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6450.pdf

Summary

The WMT18 Metrics Shared Task measures how well automatic machine translation metrics can stand in for human judgements of translation quality. Organizers gave participants outputs from systems in the WMT18 News Translation Task on newstest2018, covering 14 language pairs: English paired in both directions with Chinese, Czech, German, Estonian, Finnish, Russian and Turkish. Participating metrics scored entire test sets (system level), individual sentences (segment level), or both. Direct Assessment (DA) was the only human gold standard. System-level evaluation used Pearson correlation with DA, and thousands of hybrid systems made the comparisons more robust. Because sentence-level DA data were sparse, the organizers converted DA scores into pairwise preferences (daRR) and scored metrics with a Kendall's Tau-like statistic. Embedding-based metrics led the field: RUSE and YiSi were top performers. This differs from 2017, when character-level metrics dominated. The task gives MT developers a regularly updated, evidence-based guide to choosing evaluation metrics.

Key points

  • Ten metrics submitted by 8 research groups were benchmarked against 8 baselines, namely CDER, TER, WER, PER, chrF, NIST, SentBLEU and BLEU.
  • The data comprised 149 MT systems across 14 language pairs, plus 10K hybrid systems per pair built by randomly mixing sentences from two real systems. newstest2018 has about 3,000 sentences per direction, except Chinese (3,981) and Estonian (2,000).
  • For daRR, two translations of the same source formed a preference pair only if their DA scores differed by more than 25 points.
  • At system level, correlations often exceeded 0.9, with the best metrics passing 0.95 on several pairs. Some pairs dropped to 0.7–0.8. Averaged across languages, NIST ranked slightly above RUSE.
  • Segment-level Kendall's τ mostly fell between 0.25 and 0.4, with a maximum of about 0.69. RUSE was strongest for into-English and YiSi-1 for out-of-English. YiSi, BEER, CharacTER, RUSE and BLEND all beat sentBLEU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

76 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 62 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6450

@inproceedings{ma2018results,
  title = {{Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance}},
  author = {Ma, Qingsong and Bojar, Ondřej and Graham, Yvette},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6450},
  url = {https://doi.org/10.18653/v1/w18-6450}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A Grain of Salt for the WMT Manual Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  2. Ondřej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016. Ten Years of WMT Evaluation Campaigns: Lessons Learnt. In Proceedings of the LREC 2016 Workshop “Translation Evaluation – From Fragmented Tools and Data Sets to an Integrated Ecosystem”, pages 27–34, Portorose, Slovenia, 5. find
  3. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. Linguistics, pages 443–451, Gothenburg, Sweden, April. Association for Computational Linguistics. find our card
  4. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014b. Randomized significance tests in machine translation. In Proceedings of the ACL 2014 Ninth Workshop on Statistical Machine Translation, pages 266–274. Association for Computational Linguistics. find
  5. Ondřej Bojar, Jiří Mírovský, Kateřina Rysová, and Magdaléna Rysová. 2018. Evald reference-less discourse evaluation for wmt18. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels, October. Association for Computational Linguistics. find
  6. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate Evaluation of Segment-level Machine Translation Metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, Denver, Colorado. find
  7. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland, USA, June. Association for Computational Linguistics. find our card
  8. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using Ngram Co-occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28, 1. find
  10. Yinuo Guo, Chong Ruan, and Junfeng Hu. 2018. Meteor++: Incorporating copy knowledge into machine translation evaluation. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels, October. Association for Computational Linguistics. find
  11. Melania Duma and Wolfgang Menzel. 2017. UHH submission to the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  12. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation Between European Languages. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’06, pages 102–121, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  13. Yvette Graham and Timothy Baldwin. 2014. Testing for Significance of Increased Correlation with Human Judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar, October. Association for Computational Linguistics. find
  14. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  15. Yvette Graham and Qun Liu. 2016. Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA. Association for Computational Linguistics. find
  16. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In In Proceedings of EACL, pages 241–248. find
  17. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  18. Chi-kiu Lo. 2018. The NRC metric submission to the WMT18 metric and parallel corpus filtering shared task. In Arxiv. find
  19. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined MT metric based on direct assessment — casict-dcu submission to WMT17 metrics task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  20. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014a. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD, USA. Association for Computational Linguistics. for the evaluation of text-to-speech systems. In Proc. of ICSLP - INTERSPEECH. find our card
  21. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016a. Character: Translation edit rate on character level. In ACL 2016 First Conference on Machine Translation, pages 505–510, Berlin, Germany, August. find our card
  22. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  23. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016b. Charac-Ter: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation, Berlin, Germany, August. Association for Computational Linguistics. find
  24. Joybrata Panja and Sudip Kumar Naskar. 2018. Iter: Improving translation edit rate through optimizable edit costs. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels, October. Association for Computational Linguistics. find
  25. Evan James Williams. 1959. Regression analysis, volume 14. Wiley New York. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. find our card
  27. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find
  28. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  29. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels, October. Association for Computational Linguistics. find our card
  30. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  31. Miloš Stanojević and Khalil Sima’an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics. find
  32. Yolanda Vazquez-Alvarez and Mark Huckvale. 2002. The reliability of the ITU-t p.85 standard find