Elamur Research
☰
home / papers / 2019 / Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5302.pdf

Summary

The WMT19 Metrics Shared Task asks how reliably automatic MT evaluation metrics reflect human judgements of translation quality. Participants scored outputs from the WMT19 News Translation Task (newstest2019): 233 systems across 18 language pairs with 10 target languages. Thirteen teams submitted 24 metrics. Ten of these were reference-free and entered the "QE as a Metric" track, run jointly with the Quality Estimation Task. The organizers also computed 11 baselines.

Direct Assessment (DA) was the only human evaluation used. System-level metrics were compared against averaged, annotator-standardized DA scores. Segment-level metrics were compared against relative rankings derived from DA, called daRR.

The best metrics reached system-level Pearson correlations above 0.95 for several pairs. These correlations fell sharply when only the top-performing systems were evaluated, so results based on mixed-quality system pools can overstate how well metrics judge strong systems. Embedding-based metrics (YiSi, ESIM) again performed best. The findings show that segment-level scoring and the evaluation of strong systems remain open problems.

Key points

  • Baselines: the organizers used chrF, CDER, TER, PER, WER, NIST, and both BLEU and its sentence-level form SentBLEU, along with three reimplementations: chrF+, sacreBLEU-BLEU and sacreBLEU-chrF.
  • Test sets: newstest2019 has about 2,000 sentences per direction. Exceptions are Gujarati, Kazakh and Lithuanian (about 1,000 each) and German↔French (1,701).
  • daRR construction: a pair of translations counts as a ranking only if their DA scores differ by at least 25 points. fr-de produced just 1,369 such pairs.
  • Segment-level results: Kendall's τ ranged from 0.03 to 0.59 for reference-based metrics and from −0.153 to 0.351 for QE systems. YiSi-1_srl won every pair it entered, and ESIM won 6 of 18 pairs.
  • QE systems: system-level correlations were unstable, ranging from 0.028 to 0.947 (the latter for zh-en). UNI alone spanned 0.028 to 0.930 across language pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5302

@inproceedings{ma2019results,
  title = {{Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges}},
  author = {Ma, Qingsong and Wei, Johnny and Bojar, Ondřej and Graham, Yvette},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)},
  year = {2019},
  doi = {10.18653/v1/w19-5302},
  url = {https://doi.org/10.18653/v1/w19-5302}
}

References

38 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 597–604, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translation (W… find our card
  3. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using Ngram Co-occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  4. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  5. Ondřej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016. Ten Years of WMT Evaluation Campaigns: Lessons Learnt. In Proceedings of the LREC 2016 Workshop “Translation Evaluation Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian… find
  6. Aaron L.-F. Han, Derek F. Wong, and Lidia S. Chao. 2012. Lepor: A robust evaluation metric for machine translation with augmented factors. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pages 441–450. Association for Computational Linguistics. find
  7. Yvette Graham and Timothy Baldwin. 2014. Testing for Significance of Increased Correlation with Human Judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. find
  8. Aaron L.-F. Han, Derek F. Wong, Lidia S. Chao, Liangye He, Yi Lu, Junwen Xing, and Xiaodong Zeng. 2013. Language-independent model for machine translation evaluation with reinforced factors. In Machine Translation Summit XIV, pages 215–222. International Association for Machine Translation. find
  9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational L… find
  10. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  11. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014a. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Gothenburg, Sweden. Association for Computational… find
  12. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation Between European Languages. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’06, pages 102–121, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28. find
  14. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2003. A novel string-to-string distance measure with applications to machine translation evaluation. In Proceedings of Mt Summit IX, pages 240–247. find
  15. Yvette Graham and Qun Liu. 2016. Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA. Association… find
  16. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT Evaluation Using Block Movements. In In Proceedings of EACL, pages 241–248. find
  17. Chi-kiu Lo. 2019. YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find our card
  18. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014b. Randomized significance tests in machine translation. In Proceedings of the ACL 2014 Ninth Workshop on Statistical Machine Translation, pages 266–274. Association for Computational Linguistics. find
  19. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguis… find our card
  20. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate Evaluation of Segment-level Machine Translation Metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, Denver, Colorado. find
  21. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD, USA. Association for Computational Linguistics. find our card
  22. Yinuo Guo and Junfeng Hu. 2019. Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  23. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics. find
  24. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016a. Character: Translation edit rate on character level. In ACL 2016 First Conference on Machine Translation, pages 505–510, Berlin, Germany. find our card
  25. Nitika Mathur, Tim Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proc. of ACL (short papers). To appear. find
  26. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016b. Charac-Ter: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. find our card
  28. Evan James Williams. 1959. Regression analysis, volume 14. Wiley New York. find
  29. Elizaveta Yankovskaya, Andre Tättar, and Mark Fishel. 2019. Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  30. Maja Popovic. 2012. Morpheme- and POS-based IBM1 and language model scores for translation quality estimation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT@NAACL-HLT 2012, June 7-8, 2012, Montréal, Canada, pages 133–137. find
  31. Ryoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, and Mamoru Komachi. 2019. Filtering Pseudo-References by Paraphrasing for Automatic Evaluation of Machine Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computationa… find
  32. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal. Association for Computational Linguistics. find
  33. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  34. Matt Post. 2018a. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics. find our card
  35. Matt Post. 2018b. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find our card
  36. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  37. Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019. EED: Extended Edit Distance Measure for Machine Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  38. Miloš Stanojević and Khalil Sima’an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal. Association for Computational Linguistics. find