Elamur Research
☰
home / papers / 2020 / Unbabel's Participation in the WMT20 Metrics Shared Task

Unbabel's Participation in the WMT20 Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.wmt-1.101.pdf

Summary

This paper presents Unbabel's submission to the WMT 2020 Metrics Shared Task. It targets the segment-, document- and system-level tracks and the "QE as a Metric" track. The systems are built on the COMET framework. A cross-lingual XLM-RoBERTa encoder embeds the source, MT hypothesis and reference, pooling turns these into sentence vectors, and a feed-forward head predicts quality.

The team trained three Estimator models on different kinds of human judgement: HTER, MQM and Direct Assessment (DA). They also trained a COMET-RANK model on relative rankings derived from DA. The authors add three techniques:

  • combining several references at inference time,
  • turning segment scores into a document score with a length-weighted average,
  • a Kendall's Tau check at system level, to test how well metrics separate strong systems that perform similarly.

Because evaluation uses WMT19 test sets, the reported numbers are not official WMT20 results. On these sets, the models match or beat BLEU, chrF, BERTScore, BLEURT and PRISM, often setting a new state of the art. The results suggest a neural metric that reads the source is a strong, practical MT evaluator. They also suggest that reference quality matters more than the number of references.

Key points

  • Estimator setup. The Estimators use XLM-RoBERTa large with larger feed-forward layers (3072 and 1536 units) and a frozen embedding layer. Learning rates shrink by a factor of 0.95 per layer going down. COMET-RANK stays on the base encoder, because larger encoders made its training unstable.
  • Training data:
    • an HTER corpus of 211K tuples from QT21 plus APE-QUEST,
    • a proprietary MQM corpus of 27K customer-support tuples, from English into 15 languages or variants,
    • WMT DA/DA RR data covering 24 language pairs, using 2017–2018 for training and 2019 for validation.
  • Document level (en-cs, en-de). Length-weighted "micro-averaging" slightly outperformed a plain mean. The HTER Estimator scored highest, with an average Pearson r of 0.607, compared with 0.598 for DA and 0.577 for MQM.
  • Multiple references. For de-en, the alternative reference alone gave r=0.34, versus 0.42 with the original reference. For en-de, extra references from Freitag et al. (2020) did not noticeably change segment-level Kendall's Tau.
  • QE as a Metric. The reference-free variant simply drops the reference input and shrinks the regressor to match.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 61 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.101

@inproceedings{rei2020unbabel,
  title = {{Unbabel's Participation in the WMT20 Metrics Shared Task}},
  author = {Rei, Ricardo and Stewart, Craig and Farinha, Ana C and Lavie, Alon},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.101},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.101}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. find our card
  2. Aljoscha Burchardt and Arle Lommel. 2014. Practical Guidelines for the Use of MQM in Scientific Research on Translation quality. Quality Translation 21. (access date: 2020-05-26). find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  4. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be Guilty but References are not Innocent. ArXiv, abs/2004.06063. ↗ our card
  5. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  7. Julia Ive, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Joachim Van den Bogaert, Eduardo Farah, Christine Maroti, Artur Ventura, and Maxim Khalilov. 2020. A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3692–3697, Marseille, France. European Language Resources Association. find
  8. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, António Góis, M. Amin Farajian, António V. Lopes, and André F. T. Martins. 2019. Unbabel’s participation in the WMT19 translation quality estimation shared task. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 78–84, Florence, Italy. Association for Computational Linguistics. find
  9. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. find
  10. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  11. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. find our card
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  13. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find our card
  14. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  15. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2019. Machine Translation Evaluation with BERT Regressor. arXiv preprint arXiv:1907.12679. ↗
  16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  17. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadina, Matteo Negri, , and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Machine Translation Summit XVI, pages 55–71, Nagoya, Japan. find
  18. Kosuke Takahashi, Katsuhito Sudoh, and Satoshi Nakamura. 2020. Automatic machine translation evaluation using source language inputs and cross-lingual language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3553–3558, Online. Association for Computational Linguistics. find
  19. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. ArXiv, abs/2004.14564. ↗ our card
  20. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find