Elamur Research
☰
home / papers / 2023 / Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.wmt-1.73.pdf

Summary

Quality Estimation (QE) scores machine translation output without a reference translation. This paper describes the Unbabel–IST entry to the WMT 2023 QE shared task. Task 1 asked for quality predictions at both the sentence and word level, while Task 2 required pinpointing individual error spans and labelling each one as MINOR or MAJOR. The team kept the COMETKIWI-22 architecture: a multilingual encoder whose layers are combined with sparsemax weights, plus a sentence-score head and a word-tag head. The main change was much larger encoders (XLM-R XL and XXL) and more training data. They first trained generic sentence-level models on about 940k scored segments in 38 language pairs, drawn from WMT DA data, MLQE-PE and this year's DA/MQM releases. These models were then adapted to each task with a weighted mix of sentence and word losses. Final submissions are Optuna-weighted ensembles. The systems ranked first in every multilingual track and beat COMETKIWI-22 by as much as 10 Spearman points. GPT-4, prompted in an AutoMQM-like way, was competitive on error spans but weak at sentence level. The released XL and XXL checkpoints are, per the authors, the largest openly available QE models.

Key points

  • Results: sentence-level reached 59.4 Spearman, almost 4 points above the runner-up. Word-level reached 31.7 MCC, about 2 points ahead. Error span detection reached 22 F1, more than 5 points ahead.
  • Encoders: InfoXLM L (550M parameters), XLM-R XL (3.5B) and XLM-R XXL (10.7B).
  • Sentence-level ensemble: two XL models fine-tuned on 2023 DA data plus two generic XXL models. The XL models carried more weight for the Indian languages (en-mr, en-hi, en-ta, en-te, en-gu), and the XXL models for the MQM pairs.
  • Word level: models trained on post-edit data performed surprisingly well on the MQM pairs. The best he-en score came from a post-edit model, even though he-en had no training data.
  • GPT-4 limitations: the authors found it hard to control GPT-4's precision and recall, which shifted with the number, order and content of the prompt examples. It was also costly and slow to run.
  • Released models: Unbabel/wmt23-cometkiwi-da-xl and Unbabel/wmt23-cometkiwi-da-xxl on Hugging Face.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.wmt-1.73

@inproceedings{rei2023scaling,
  title = {{Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task}},
  author = {Rei, Ricardo and Guerreiro, Nuno M. and Pombal, José and Stigt, Daan van and Treviso, Marcos and Coheur, Luisa and Souza, José G. C. de and Martins, André F. T.},
  booktitle = {Proceedings of the Eighth Conference on Machine Translation},
  year = {2023},
  doi = {10.18653/v1/2023.wmt-1.73},
  url = {https://doi.org/10.18653/v1/2023.wmt-1.73}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. find our card
  2. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Assoc… find our card
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  4. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, J. Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. ArXiv, abs/2308.07286. find our card
  5. Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset. In Proceedings of the Language Resources and Evaluation Conference, pages 4… find
  6. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. XCOMET: Transparent machine translation evaluation through fine-grained error detection. find our card
  7. Yuang Li, Chang Su, Ming Zhu, Xinglin Piao, Mengyao Lyu, Min Zhang, and Hao Yang. 2023. find
  8. HW-TSC 2023 Submission for the Quality Estimation Shared Task. In Proceedings of the Eighth Conference on Machine Translation, Singapore. Association for Computational Linguistics. find
  9. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623. find
  10. OpenAI. 2023. GPT-4 technical report. find our card
  11. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  12. Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. 2022a. Searching for COMETINHO: The little metric that could. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 61–70, Ghent, Belgiu… find
  13. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. find
  14. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  15. Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018. Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. find
  16. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. find
  17. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics. find
  18. Zeyu Yan, Wenbo Zhang, Qiaobo Deng, Hongbao Mao, Jie Cai, and Zhengyu He. 2023. IOL Research’s Submission for WMT 2023 Quality Estimation Shared Task. In Proceedings of the Eighth Conference on Machine Translation, Singapore. Association for Computational Linguistics. find