Elamur Research
☰
home / papers / 2022 / CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.wmt-1.60.pdf

Summary

This paper describes the joint IST and Unbabel entry to the WMT 2022 Quality Estimation shared task. Quality Estimation (QE) scores machine translation output without reference translations. The team entered all three subtasks: sentence- and word-level quality prediction, Explainable QE, and Critical Error Detection.

They built COMETKIWI, which ports the predictor-estimator design of OpenKiwi into the COMET framework and adds a word-level OK/BAD tagger. Models were first pretrained on WMT Metrics Direct Assessment data using the UniTE objective, which exploits reference translations. They were then fine-tuned on source–translation pairs, with sentence and word losses optionally optimized together. Language pairs absent from training were handled through few-shot adaptation.

For explanations, the authors combined attention weights with gradient norms and learned how much each attention head should contribute. Ensembles of these systems ranked first in the multilingual setting of every track, often by a wide margin. This shows that reference-aware pretraining, multi-task objectives and attention-gradient explanations transfer well across languages.

Key points

  • Encoders and data: three encoders were tested, XLM-R Large, InfoXLM Large and RemBERT.
    • Pretraining used WMT DA judgments from 2017–2019, covering 24 LPs and about 657k samples.
    • Fine-tuning used MLQE-PE (DA, HTER) and the WMT 2020/2021 MQM annotations.
  • Few-shot adaptation: about 500 dev examples each were used for km-en, ps-en, en-ja and en-cs. This helped those pairs without hurting the others. en-yo remained zero-shot.
  • Ensembles: the DA submission averaged six models with per-LP weights tuned via Optuna. The MQM submission combined eleven models.
  • Multilingual results:
    • 0.572 Spearman on sentence-level DA, about 7% above the runner-up.
    • 0.341 MCC on word level, about 2.4% higher.
    • 0.486 R@K on Explainable QE, about 10% higher.
  • Per-LP results and explanation method: the systems were best on 6/9 LPs at sentence level and 5/9 at word level. In Explainable QE they led on all pairs except km-en and ps-en. The new Attn × GradNorm method with Head Mix outperformed last year's Attn × Norm baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

308 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 294 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wmt-1.60

@inproceedings{rei2022cometkiwi,
  title = {{CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task}},
  author = {Rei, Ricardo and Treviso, Marcos and Guerreiro, Nuno M. and Zerva, Chrysoula and Farinha, Ana C and Maroti, Christine and Souza, José G. C. de and Glushkova, Taisiya and Alves, Duarte and Coheur, Luisa and Lavie, Alon and Martins, André F. T.},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.60},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.60}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. find our card
  2. Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. Explaining predictions of non-linear classifiers in NLP. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 1–7, Berlin, Germany. Association for Computational Linguistics. find
  3. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Assoc… find our card
  4. George Chrysostomou and Nikolaos Aletras. 2022. An empirical study on explanations in out-of-domain settings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6920–6938, Dublin, Ireland. Association for Computational Linguistic… find
  5. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking Embedding Coupling in Pre-trained Language Models. In International Conference on Learning Representations. find our card
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. 2021. The Eval4NLP shared task on explainable quality estimation: Overview and results. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165–178, Punta Cana, Dominican Republic. A… find
  9. Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset. In Proceedings of the Language Resources and Evaluation Conference, pages 4… find
  10. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  11. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Trans… find our card
  12. Nuno M. Guerreiro and André F. T. Martins. 2021. SPECTRA: Sparse structured text rationalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6534–6550, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  13. Julia Ive, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Joachim Van den Bogaert, Eduardo Farah, Christine Maroti, Artur Ventura, and Maxim Khalilov. 2020. A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In Proceedings of the 12th Language Resources… find
  14. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy. Assoc… find
  15. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational… find our card
  16. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. Association for Computational Linguistics. find
  17. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623. find
  18. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  19. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest at WMT2020: Sentence-level direct assessment. In Proceedings of the Fifth Conference on Machine Translation, pages 1049– 1055, Online. Association for Computational Linguistics. find
  20. Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. 2022. Searching for COMETINHO: The little metric that could. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 61–70, Ghent, Belgium… find
  21. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  22. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In Proc. ACM SIGKDD, pages 1135–1144. ACM. find our card
  23. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  24. Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 Shared Task on Quality Estimation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find our card
  25. Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. find
  26. IST-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 961–972, Online. Association for Computational Linguistics. find
  27. A The data used for finetuning our QE systems is shown in Table 5. For DA data, we split the original development set to generate a new dev/test split, therefore the reported numbers in the table correspond to this “internal” dev split. find
  28. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadin, Matteo Negri, and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Proceedings of Machine Translation Summit XVI: Research Track, pages 55–71, Nagoya Japa… find
  29. LP TRAIN en-de en-mr en-zh et-en ne-en ro-en ru-en si-en en-de † en-ru † zh-en † Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018. Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. find
  30. Marcos Treviso, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2021. find
  31. IST-unbabel 2021 submission for the explainable quality estimation shared task. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 133– 145, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  32. DEV en-de en-mr en-zh et-en ne-en ro-en ru-en si-en en-cs en-ja km-en ps-en en-de † en-ru † zh-en † Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Process… find
  33. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for… find our card
  34. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Data Information Samples Source Tokens Target Tokens OK / BAD Target 9000 26000 9000 9000 9000 9000 9000 9000 54681 15628 75327 147870 690516 148657… find