Elamur Research
☰
home / papers / 2022 / Findings of the WMT 2022 Shared Task on Quality Estimation

Findings of the WMT 2022 Shared Task on Quality Estimation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.wmt-1.3.pdf

Summary

This report describes the 11th edition of the WMT shared task on Quality Estimation (QE), which asks systems to judge neural machine translation output at the sentence and word levels without any reference translation. The organisers expanded the benchmark in three ways. First, they added fine-grained Multidimensional Quality Metrics (MQM) annotations for three language pairs, alongside the existing Direct Assessment and post-edit data (MLQE-PE). Second, they added low-resource languages: a large English-Marathi dataset and an English-Yoruba test set that was kept undisclosed until test time. Third, they defined three tasks: core sentence- and word-level QE, an explainability task that treats error words as rationales for sentence scores, and a redesigned critical error detection task built on synthetic errors. Eleven teams sent 991 submissions in total. Systems produced meaningful correlations even on the unseen Yoruba pair, though scores there were lower than on seen pairs. The released data, gold labels and submissions give a public benchmark for more interpretable and robust QE, including for under-resourced languages.

Key points

  • Data sources: Training pairs come from MLQE-PE (En-De, En-Zh, Ru-En, Ro-En, Ne-En, Et-En, Si-En). Development and test pairs include En-Cs, En-Ja, Km-En and Ps-En. Translations were produced with a fairseq Transformer and MBART50.
  • English-Marathi: The source text covers healthcare (NHS corpus), cultural and general/news domains, the latter two drawn from the IITB English-Hindi Parallel Corpus. It was translated with IndicTrans, which was trained on Samanantar.
  • English-Yoruba zero-shot set: It contains 1010 Wikipedia sentences on 7 topics, translated with Google Translate. Annotator agreement was 0.487 Pearson at sentence level and 0.380 kappa at word level. Results depended on whether the pretrained encoder had seen Yoruba.
  • Critical error detection corpus: Built with SMAUG, it simulates additions, deletions, named-entity errors, polarity changes and number errors. The task ran in constrained and unconstrained settings.
  • Submission rules: Submissions went through CodaLab, limited to 2 per day and 10 per task and language pair. All data is published at github.com/WMT-QE-Task/wmt-qe-2022-data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

89 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 83 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wmt-1.3

@inproceedings{zerva2022findings,
  title = {{Findings of the WMT 2022 Shared Task on Quality Estimation}},
  author = {Zerva, Chrysoula and Blain, Frédéric and Rei, Ricardo and Lertvittayakumjorn, Piyawat and Souza, José G. C. de and Eger, Steffen and Kanojia, Diptesh and Alves, Duarte and Orăsan, Constantin and Fomicheva, Marina and Martins, André F. T. and Specia, Lucia},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.3},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.3}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021. Toward a perspectivist turn in ground truthing for predictive computing. arXiv preprint arXiv:2109.04270. ↗ our card
  2. Hervé Abdi. 2007. The bonferroni and šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics, 3:103–107. find
  3. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online. Association for Computational Linguistics. human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  4. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo George Foster, Craig Stewart, Tom Kocmi, Eleftherios Avramidis, Alon Lavie, and André F. T. Martins. 2022. Results of the WMT22 Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  5. George Chrysostomou and Nikolaos Aletras. 2022. An empirical study on explanations in out-of-domain settings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6920–6938, Dublin, Ireland. Association for Computational Linguistics. find
  6. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. find our card
  7. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking Embedding Coupling in Pre-trained Language Models. In International Conference on Learning Representations. find our card
  8. Xiang Geng, Yu Zhang, Shujian Huang, Shimin Tao, Hao Yang, and Jiajun Chen. 2022. NJUNLP’s Participation for the WMT2022 Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  9. Qu Cui, Shujian Huang, Jiahuan Li, Xiang Geng, Zaixiang Zheng, Guoping Huang, and Jiajun Chen. 2021. Directqe: Direct pretraining for machine translation quality estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12719–12727. find
  10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28. find
  11. David M Eberhard, Gary F Simons, and Charles D Fennig. 2020. Ethnologue: Languages of the world (2020). URL: https://www.ethnologue. com/(visited on Apr. 11, 2020)(cit. on p. 14). ↗
  12. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098–6111, Hong Kong, China. Association for Computational Linguistics. find our card
  13. Sugyeong Eo, Chanjun Park, Hyeonseok Moon, Jaehyung Seo, and Heuiseok Lim. 2022. KU X Upstage’s submission for the WMT22 Quality Estimation: Critical Error Detection Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  14. Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. 2021. The Eval4NLP shared task on explainable quality estimation: Overview and results. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165–178, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  15. Hui Huang, Hui Di, Chunyou Li, Hanming Wu, Kazushige Oushi, Yufeng Chen, Jian Liu, and Jin’an Xu. 2022. BJTU-Toshiba’s Submission to WMT22 Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  16. Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A multilingual quality estimation and post-editing dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4963–4974, Marseille, France. European Language Resources Association. find
  17. Diptesh Kanojia, Marina Fomicheva, Tharindu Ranasinghe, Frédéric Blain, Constantin Orăsan, and Lucia Specia. 2021. Pushing the right buttons: Adversarial evaluation of quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 625–638, Online. Association for Computational Linguistics. find
  18. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of Demonstrations, pages 117–122, Florence, Italy. Association for Computational Linguistics. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. find our card
  19. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 562–568. find
  20. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  21. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  22. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find our card
  23. Seunghyun S. Lim and Jeonghyeok Park. 2022. Papago’s submission to the wmt22 quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  24. Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. Using a new analytic measure for the annotation and analysis of mt errors on real data. In Proceedings of the 17th Annual conference of the European Association for Machine Translation, pages 165–172. find
  25. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135– 1144. find our card
  26. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, USA. find
  27. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231. find our card
  28. Nikita Mehandru, Marine Carpuat, and Niloufar Selehi. 2022. Quality Estimation by Backtranslation at the WMT 2022 Quality Estimation Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  29. Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. 2021. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684–725, Online. Association for Computational Linguistics. find our card
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  31. Chang Su, Miaomiao Ma, Shimin Tao, Hao Yang, Min Zhang, Xiang Geng, Shujian Huang, Jiaxin Guo, Minghan Wang, and Yinglu Li. 2022. CrossQE: HW-TSC 2022 Submission for the Quality Estimation Shared. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  32. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022. Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages. Transactions of the Association for Computational Linguistics, 10:145–162. find our card
  33. Y. Tang, C. Tran, Xian Li, P. Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401. ↗
  34. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. find
  35. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics. find our card
  36. Jiayi Wang, Ke Wang, Boxing Chen, Yu Zhao, Weihua Luo, and Yuqi Zhang. 2021. QEMind: Alibaba’s submission to the WMT21 quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 948–954, Online. Association for Computational Linguistics. find
  37. Evan J. Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA. find
  38. Alexander Yeh. 2000. More Accurate Tests for the Statistical Significance of Result Differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953, Saarbrücken, Germany. find
  39. Eirini Zafeiridou and Sokratis Sofianopoulos. 2022. Welocalize-ARC/NKUA’s Submission to the WMT 2022 Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find