Elamur Research
☰
home / papers / 2021 / Findings of the WMT 2021 Shared Task on Quality Estimation

Findings of the WMT 2021 Shared Task on Quality Estimation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.wmt-1.71.pdf

Summary

Quality Estimation (QE) predicts how good a machine translation is without access to a reference translation. The 2021 edition of the WMT QE shared task ran three subtasks on output from neural MT systems:

  • Task 1: sentence-level direct assessment (DA) scores.
  • Task 2: post-editing effort, as sentence-level HTER plus word-level OK/BAD tags.
  • Task 3: a new binary task that flags translations containing critical errors.

The organisers extended the MLQE-PE benchmark with fresh test sets and additional post-edited data. They also added four language pairs with no training data at all, to test zero-shot prediction. Submissions ran on CodaLab, and for the first time the evaluation reported model size alongside accuracy.

Zero-shot results were unexpectedly strong and roughly on par with supervised pairs, except for En-Ja. The organisers credit this mainly to fine-tuning on synthetic data. Critical error detection gave encouraging MCC scores.

The task matters because it benchmarks multilingual, low-resource and deployment-friendly QE. Its focus on errors that can mislead readers is especially relevant when MT is used without human editing. Gold labels and all submissions were released publicly.

Key points

  • Participation: 19 institutions submitted 1,263 systems in total. Each team could submit up to 30 systems per task and language pair, or 10 for the multilingual track.
  • Task 1 data:
    • Seven pairs with 7,000/1,000/1,000 train/dev/test sentences each: En-De, En-Zh, Ru-En, Ro-En, Et-En, Si-En, Ne-En.
    • Translations came from fairseq Transformer models.
    • Zero-shot test sets for En-Cs, En-Ja, Ps-En and Km-En were translated with the ML50 fairseq multilingual model.
  • Task 2 labels: HTER was computed with TERCOM. Word-level tags cover source words, target words and the gaps between them, so omissions can be captured.
  • Task 3 (critical errors):
    • Language pairs: En-Cs, En-De, En-Ja and En-Zh.
    • Error categories: TOX, SAF, NAM, SEN and NUM.
    • Results were strongest for En-Cs and En-De in MCC. Reported F1 at detecting errors was around 0.9 or above.
  • Comparison with 2020: the best Pearson correlations rose for En-De (0.55→0.58), En-Zh (0.54→0.60) and Ne-En (0.82→0.87), but fell for Si-En (0.68→0.61).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

80 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{specia2021findings,
  title = {{Findings of the WMT 2021 Shared Task on Quality Estimation}},
  author = {Specia, Lucia and Blain, Frédéric and Fomicheva, Marina and Zerva, Chrysoula and Li, Zhenhao and Chaudhary, Vishrav and Martins, André F. T.},
  booktitle = {Proceedings of the Sixth Conference on Machine Translation},
  year = {2021},
  doi = {10.48448/n1nv-gd47},
  url = {https://doi.org/10.48448/n1nv-gd47}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ergun Biçici. 2021. Rtm superlearner results at quality estimation task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  2. Shaika Chowdhury, Naouel Baili, and Brian Vannah. 2021. Ensemble fine-tuned mbert for wmt21 translation quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  3. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transportation distances. arXiv preprint arXiv:1306.0895. ↗
  4. Shuoyang Ding, Marcin Junczys-Dowmunt, Matt Post, Christian Federmann, and Philipp Koehn. 2021. The jhu-microsoft submission for wmt21 quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  5. Herbert Edelsbrunner. 2013. Persistent homology: theory and practice. find
  6. Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2020. MLQE-PE: A multilingual quality estimation and post-editing dataset. arXiv preprint arXiv:2010.04480. Marina Fomicheva and Lucia Specia were supported by funding from the Bergamot project (EU Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Obamuyide, H2020 Grant No. 825303). ↗
  7. André Martins and Nikolaos Aletras, and Lucia Specia. 2021. Knowl-Chrysoula Zerva were funded by the P2020 pro-edge distillation for quality estimation. grams Unbabel4EU (contract 042671) and MAIA contract 045909), by the European Research Coun- Gregor Geigle, Jonas Elias Stadtmüller, Wei Zhao, Jonas Pfeiffer, and Steffen Eger. 2021. Tuda cil (ERC StG DeepSPIN 758969), and by the Fun-at wmt21: Sentence-level direct assessment with dação para a Ciência e Tecnologia through contract adapters. In Proceedings of the Sixth Conference on UIDB/50008/2020. We would like to thank Er-Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. ick Fonseca for answering questions about data 701 find
  8. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098–6111, Hong Kong, China. Association for Computational Linguistics. find our card
  9. Raphaël Rubino, Atsushi Fujita, and Benjamin Marie. 2021. Nict kyoto submission for the wmt’21 quality estimation task: Multimetric multilingual pre-training for critical error detection. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  10. Dam Heo, WonKee Lee, Baikjin Jung, and Jong-Hyeok Lee. 2021. Quality estimation using dual encoders with transfer learning. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  11. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231. find our card
  12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. find
  13. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the wmt 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 743–764, Online. Association for Computational Linguistics. find our card
  14. Genze Jiang, Zhenhao Li, and Lucia Specia. 2021. Icl’s submission to the wmt21 critical error detection shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  15. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of ACL 2019 System Demonstrations. find
  16. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 562–568. find
  17. Seunghyun Shaun Lim, Hantae Kim, and Hyunjoong Kim. 2021. Papago submission for the wmt21 quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  18. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. find
  19. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL. ↗
  20. Y. Tang, C. Tran, Xian Li, P. Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401. ↗
  21. Ivan Vulić, Sebastian Ruder, and Anders Søgaard. 2020. Are all good word vector spaces isomorphic? arXiv preprint arXiv:2004.04070. ↗
  22. Jiayi Wang, Ke Wang, Boxing Chen, Yu Zhao, Weihua Luo, and Yuqi Zhang. 2021. Qemind: Alibaba’s submission to the wmt21 quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  23. Evan J. Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA. find
  24. Lisa Yankovskaya and Mark Fishel. 2021. Bergamot-utartu participation in the wmt’2021 quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find
  25. Alexander Yeh. 2000a. More Accurate Tests for the Statistical Significance of Result Differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953, Saarbrücken, Germany. find
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. find our card
  27. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation quality estimation with cross-lingual transformers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5070–5081, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  28. Alexander Yeh. 2000b. More accurate tests for the statistical significance of result differences. arXiv preprint cs/0008005. find
  29. Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro G. Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. Ist-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation Shared Tasks Papers, Online. Association for Computational Linguistics. find