Elamur Research
☰
home / papers / 2022 / Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust

Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust

What it is about

The shared task scored submitted MT metrics on WMT22 system outputs for three pairs (English to German, English to Russian, Chinese to English) across four domains: news, social, e-commerce and chat. The human ratings were expert MQM annotations rather than crowd-sourced DA, because DA was shown to be unreliable for high-quality output. Five extra systems were added (MBR decoding and reranking) to test metrics on output stronger than the WMT submissions. Each metric was ranked by its average rank over 201 tasks, with system-level pairwise accuracy weighted at a quarter and the rest split by a hierarchical scheme, so that segment-level tasks do not count three times. Neural fine-tuned metrics ranked top; BLEU and its variants ranked near the bottom, and the four challenge sets confirmed the same picture. Against the crowd-sourced ratings of the general task, the n-gram metrics also ranked poorly.

Ideas

For Mova

The measurement layer is tools/translate (measure.rs), and the author deltas it ranks are open problems 6 and 8 (archaic English as a delta, style classes). The neural metrics are not for Mova, but the averaging scheme over many tasks and the error weighting are count-based and can be written as tables in Rust. Nothing here is a claim about Mova until two identical runs agree.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.wmt-1.2.pdf

Summary

The WMT22 Metrics Shared Task examines how well automatic MT evaluation metrics agree with human judgments and whether string-overlap scores like BLEU remain adequate. Participants scored outputs of the WMT22 general MT task, whose test sets contain about 2,000 segments per direction spread over four domains: news, social, e-commerce and conversational. For en→de, en→ru and zh→en, the organizers did not rely on crowd-style DA scores. Instead, they gathered MQM error annotations from professional translators or trained raters. They also enlarged the system pool with M2M100 1.2B, MBR-decoded outputs and QE-reranked outputs, which tend to be hard for metrics. Both reference-based and reference-free (QE) metrics were included. A new challenge-set subtask asked participants to build contrastive tests that probe specific error types. Metrics were ranked by their average rank over 201 evaluation scenarios. Learned neural metrics clearly beat lexical ones and stayed robust across domains, which is strong evidence that the field should stop using BLEU as its default.

Key points

  • Top and bottom of the ranking: METRICX XXL ranked first (average rank 1.20), followed by COMET-22 (1.32), UNITE and BLEURT-20. Among the 13 reference-based metrics, BLEU came last, with f200spBLEU and chrF just above it.
  • Ranking method: Each metric's average rank across 201 tasks determined the final order. Last year's pairwise system-ranking accuracy was kept as one single task carrying 25% of the total weight.
  • Extra systems: The added outputs included MBR runs using BLEU, BLEURT-20 and COMET-20 as utility functions. These were built on a deliberately modest transformer-big model, sampling 256 candidates per segment.
  • Challenge sets: Four challenge sets were submitted. DFKI's covered roughly 20,000 de↔en items from 145 systems, spanning 100+ phenomena. HWTSC's contained 721 zh-en examples across five critical error types. Named entities, numbers and units proved hard for many metrics.
  • Field-level trends: For the first time since 2008, no new purely lexical metric was submitted. All scores were computed with the MTME toolkit, and the conclusions were confirmed with DA+SQM judgments.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wmt-1.2

@inproceedings{freitag2022results,
  title = {{Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust}},
  author = {Freitag, Markus and Rei, Ricardo and Mathur, Nitika and Lo, Chi-kiu and Stewart, Craig and Avramidis, Eleftherios and Kocmi, Tom and Foster, George and Lavie, Alon and Martins, André F. T.},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.2},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.2}
}

References

40 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duarte M. Alves, Ricardo Rei, Ana C. Farinha, José G. C. de Souza, and André F. T. Martins. 2022. Robust MT evaluation with Sentence-level Multilingual data Augmentation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  2. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(1). find
  3. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. find our card
  4. Chantal Amrhein, Nikita Moghe, and Liane K. Guillou. 2022. ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find our card
  5. Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1396–1412, Seattle, United States. Association for Computational Linguistics. find
  6. Eleftherios Avramidis and Vivien Macketanz. 2022. Linguistically motivated evaluation of machine translation metrics based on a challenge set. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  7. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  8. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811–825. Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. find
  9. Yilun Liu, Xiaosong Qiao, Zhanglin Wu, Su Chang, Min Zhang, Yanqing Zhao, shimin tao Song Peng, Hao Yang, Ying Qin, Jiaxin Guo, Minghan Wang, Yinglu Li, Peng Li, and Xiaofeng Zhao. 2022. Partial Could Be Better Than Whole: HW-TSC 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  10. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. find our card
  11. Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. Association for Computational Linguistics. find our card
  12. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. find
  13. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM) : A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, pages 0455–463. find
  14. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  15. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, He Wang, Renlong Ai, Shushen Manakhimova, Ursula Strohriegel, Sebastian Möller, and Hans Uszkoreit. 2022. A linguistically motivated test suite to semi-automatically evaluate German–English machine translation output. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 936–947, Marseille, France. European Language Resources Association. find
  16. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  17. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, Maja Popović, and Mariya Shmatova. 2022a. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation. Association for Computational Linguistics. find
  18. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997. find our card
  19. Ananya Mukherjee, Hema Ala, Manish Shrivastava, and Dipti Misra Sharma. 2020. Mee: an automatic metric for evaluation using embeddings for machine translation. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 292–299. IEEE. find
  20. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. find
  21. Ananya Mukherjee and Manish Shrivastava. 2022a. REUSE: REference-free UnSupervised quality Estimation Metric. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  22. Tom Kocmi, Hitokazu Matsushita, and Christian Federmann. 2022b. MS-COMET: Larger Filtered Human Annotations Help Metric Performance. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  23. Ananya Mukherjee and Manish Shrivastava. 2022b. Unsupervised Embedding-based Metric for MT Evaluation with Improved Human Correlation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  24. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Mathias Müller and Rico Sennrich. 2021. Understanding the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 259–272, Online. Association for Computational Linguistics. find our card
  25. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find our card
  26. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Florence, Italy. Association for Computational Linguistics. find our card
  27. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. find our card
  28. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. find
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  30. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  31. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from denoising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466, Online. Association for Computational Linguistics. find our card
  32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  33. Yu Wan, Keqin Bao, Dayiheng Liu, Baosong Yang, Derek F. Wong, Lidia S. Chao, Wenqiang Lei, and Jun Xie. 2022a. Alibaba-Translate China’s Submission for WMT2022 Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  34. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. Machine Translation Evaluation as a Sequence Tagging Problem. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  35. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  36. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022b. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics. find our card
  37. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. find our card
  38. Wenda Xu, Yilin Tuan, Yujie Lu, Michael Saxon, Lei Li, and William Yang Wang. 2022. Not all errors are equal: Learning text generation metrics using stratified error synthesis. arXiv preprint arXiv:2210.05035. ↗
  39. Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 751–762, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. Task Human Translation Included metricx_xl_DA_2019 metricx_xxl_DA_2019 metricx_xxl_MQM_2020 BLEURT-20 metricx_xl_MQM_2020 COMET-22 COMET-20 UniTE MS-COMET-22 UniTE-ref MATESE YiSi-1 MEE4 COMETKiwi* HuaweiTSC_EE_BERTScore_0.8_With_Human HuaweiTSC_EE_BERTScore_0.8_Without_Human Cross-QE* HuaweiTSC_EE_BERTScore_0.5_With_Human COMET-QE* HuaweiTSC_EE_BERTScore_0.5_Without_Human BERTScore HuaweiTSC_EE_BERTScore_0.3_With_Human UniTE-src* MEE2 MS-COMET-QE-22* MATESE-QE* MEE f101spBLEU f200spBLEU HuaweiTSC_EE_BERTScore_0.3_Without_Human chrF BLEU HWTSC-TLM* HWTSC-Teacher-Sim* KG-BERTScore* REUSE* SEScore Accuracy No en-de Yes en-de No en-ru No zh-en Yes zh-en No find