Elamur Research
☰
home / papers / 2021 / Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task

Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.wmt-1.111.pdf

Summary

This WMT 2021 Metrics Shared Task submission from Unbabel and IST addresses the switch from Direct Assessments (DA) to MQM as the official human judgement. It adapts COMET so its scores align better with MQM. The training recipe has two stages. Models are first pre-trained for a single epoch on WMT DA data from 2015–2020, then fine-tuned for one more epoch on z-normalized MQM scores for en-de and zh-en from Freitag et al. (2021). Ensembles combine five seeds through weight averaging.

The team also introduces two further systems:

  • COMETINHO, a compact variant built on MiniLMv2.
  • An OpenKiwi quality estimation model that learns sentence scores together with word-level OK/BAD tags derived from MQM error spans.

MQM fine-tuning clearly lifts correlations above the 2020 COMET and standard baselines. The reference-free COMET-QE-MQM nearly matches the reference-based COMET-MQM, and on en-de it slightly exceeds it. This suggests reference translations may matter less than assumed, which is relevant for evaluation scenarios without human references.

Key points

  • Pre-training corpus: DA tuples (795,269) covering 33 language pairs, including en-ta. The 2020 en-de and zh-en DA data were held out to avoid overlap with the MQM set.
  • MQM data was split by document, 80/20. This gave 11,230 en-de and 15,600 zh-en training samples, with 2,950 and 4,400 segments for testing.
  • Average segment-level Pearson/Kendall on the test split:
    • COMET-MQM (2021): 0.602 / 0.454
    • COMET-QE-MQM: 0.595 / 0.445
    • COMET-DA (2020): 0.481 / 0.359
    • BLEU: 0.129 / 0.140
  • COMETINHO replaces XLM-R large with the distilled model nreimers/mMiniLMv2-L6-H384-distilled-from-XLMR-Large. It runs about 19x faster on CPU and 14.3x faster on GPU, and needs about 5x less disk space. Its MQM variant (avg Pearson 0.490) still outperforms COMET-DA (2020).
  • The OpenKiwi QE model jointly encodes source and hypothesis and is trained multitask, combining regression with sequence labelling. Its training data is 1.1M proprietary MQM-annotated pairs from the customer support domain, spanning 38 language pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

85 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rei2021are,
  title = {{Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task}},
  author = {Rei, Ricardo and Farinha, Ana C and Zerva, Chrysoula and Stigt, Daan van and Stewart, Craig and Ramos, Pedro and Glushkova, Taisiya and Martins, André F. T. and Lavie, Alon},
  booktitle = {Proceedings of the Sixth Conference on Machine Translation},
  year = {2021},
  doi = {10.48448/wrjt-a678},
  url = {https://doi.org/10.48448/wrjt-a678}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In this paper we present the Unbabel-IST’s contribution to the WMT 2021 Metrics shared task which for the first time, introduced evaluation using MQM. Our specific contributions include; the fine-tuning of Direct Assessment based models on MQM data which yields impressive gains on the described test sets and a new, lightweight C OMET model which achieves comparable performance to its predecessors. Such a light model can provide interesting opportunities for future work into the incorporation of modern metrics into MT training. Finally, but perhaps our most important contributions; we further validate the observations in (Kocmi et al., 2021) that QE as a metric is becoming competitive as an alternative to reference-based evaluation, and, we show that a word-level QE system can be successfully trained on MQM annotations and be competitive with current trainable metrics while providing some intuition about “what” is wrong with a specific translation. find
  2. Classic n-gram matching MT evaluation metrics such as B LEU (Papineni et al., 2002) have been adopted by the MT community as a primary form of MT evaluation, yet, in the recent years of the WMT Metrics shared task (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020) these classic metrics have been outperformed first by embedding-based alternatives and more recently by trainable metrics based on pre-trained models. With the rise of word embeddings (Pennington et al., 2014; Peters et al., 2018; Devlin et al., 2019), metrics such as B LEU 2 VEC (Tättar and Fishel, 2017) and M EANT 2.0 (Lo, 2017) replaced the typical word/n-gram matching by fuzzy matches based Acknowledgments on distributional word representations. These metrics appeared for the first time at the WMT Metrics We are grateful to Fabio Kepler and José G. C. de task in 2017 with M EANT 2.0- SRL achieving the Souza for their valuable feedback and discussions. highest results at segment-level. In 2018 and 2019 This work was supported by the P2020 programs Y I S I -1 (Lo, 2019), a successor of M EANT 2.0 (Lo, MAIA (contract 045909) and Unbabel4EU (con-2017), was among the winners of the WMT Metrics tract 042671), by the European Research Council task. Y I S I -1 (Lo, 2019) mostly takes advantage of (ERC StG DeepSPIN 758969), and by the Fun-BERT embeddings (Devlin et al., 2019) to create dação para a Ciência e Tecnologia through contract soft alignments between hypothesis and reference. UIDB/50008/2020. Trainable metrics started as simple regressions based on lexical features (e.g B LEND (Ma et al., 2017)) but nowadays these metrics also use em- References beddings to extract features that are then used to Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. regress on quality assessments. The first of such Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine metrics were R USE (Shimanaka et al., 2018) and Translation, pages 489–513, Copenhagen, Denmark. find our card
  3. E SIM (Mathur et al., 2019) which were based on Association for Computational Linguistics. RNN encoders and worked mostly for English. In 2020, B LEURT (Sellam et al., 2020) and C OMET Alexis Conneau, Kartikay Khandelwal, Naman Goyal, (Rei et al., 2020a) were proposed. Both metrics Vishrav Chaudhary, Guillaume Wenzek, Francisco 1035 find
  4. Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. CoRR, abs/2104.14478. ↗ our card
  7. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics. find our card
  8. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. 2021. Uncertainty-Aware Machine Translation Evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Online. Association for Computational Linguistics. find
  9. Yvette Graham, Timothy Baldwin, A. Moffat, and J. Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  10. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  11. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy. Association for Computational Linguistics. find
  12. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. CoRR, abs/2107.10821. ↗
  13. Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, pages 589–597, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  14. Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. Association for Computational Linguistics. find our card
  15. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. find
  16. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational Linguistics. find our card
  17. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined MT metric based on direct assessment — CASICT-DCU submission to WMT17 metrics task. In Proceedings of the Second Conference on Machine Translation, pages 598–603, Copenhagen, Denmark. Association for Computational Linguistics. find
  18. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  19. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. Association for Computational Linguistics. find
  20. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  23. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  24. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  25. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020. Minilmv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. CoRR, abs/2012.15828. ↗
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  28. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics. find our card
  30. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  31. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 751–758, Belgium, Brussels. Association for Computational Linguistics. find our card
  32. Andre Tättar and Mark Fishel. 2017. bleu2vec: the painfully familiar metric on continuous vector space steroids. In Proceedings of the Second Conference on Machine Translation, pages 619–622, Copenhagen, Denmark. Association for Computational Linguistics. find
  33. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics. find our card