Elamur Research
☰
home / papers / 2022 / UniTE: Unified Translation Evaluation

UniTE: Unified Translation Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.acl-long.558.pdf

Summary

Machine translation evaluation usually comes in three variants: comparing a hypothesis with a reference (REF), with the source sentence only, also called quality estimation (SRC), or with both (SRC+REF). Existing metrics such as BLEURT, TransQuest and COMET each target only one of these settings. UniTE handles all three in a single model built on a multilingual PLM (XLM-R). It concatenates the hypothesis with whichever segments are available and passes them through the encoder together. A pooled representation then feeds a feedforward regressor trained with mean squared error against human scores.

Two techniques bridge the gap between two-segment PLM pretraining and three-segment inputs:

  • Monotonic Regional Attention (MRA) masks selected attention flows between segments.
  • Unified pretraining uses synthetic translations labeled through ranking-based score normalization in a multi-task setup.

On WMT 2019 Metrics, one UniTE model beats strong task-specific baselines across all three tasks. On WMT 2020 QE, it is competitive with the winning system. This suggests one evaluator can replace several specialized metrics.

Key points

  • On English-targeted WMT 2019 Metrics directions, UniTE gains about 1.1 (REF), 2.3 (SRC) and 1.1 (SRC+REF) absolute Kendall's τ points over strong baselines.
  • Two variants are proposed: UniTE-MRA, which needs no pseudo-labeled data, and UniTE-UP, which uses unified pretraining. A multilingual-target version, UniTE-MUP, leads on non-English-targeted directions and is competitive with the WMT 2020 QE winner (TransQuest).
  • In the MRA ablation, softly blocking hypothesis→source attention raises the average τ from 34.1 to 34.8. Removing reference→hypothesis attention hurts the most (−0.5).
  • Unified pretraining combined with unified finetuning gives the best scores (τ 34.8 / 30.0 / 35.0 for REF / SRC / SRC+REF), outperforming task-specific training.
  • Ranking-based labeling with an ensemble of 3 checkpoints (35.0) beats Z-score normalization (34.2 single, 33.5 ensembled). Code and models are released at github.com/NLP2CT/UniTE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.558

@inproceedings{wan2022unite,
  title = {{UniTE: Unified Translation Evaluation}},
  author = {Wan, Yu and Liu, Dayiheng and Yang, Baosong and Zhang, Haibo and Chen, Boxing and Wong, Derek and Chao, Lidia},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.558},
  url = {https://doi.org/10.18653/v1/2022.acl-long.558}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation. In Proceedings of the Third Conference on Machine Translation (WMT). find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  4. Peter Emerson. 2013. and Partial Voting. 40(2):353–358. find
  5. The Original Borda Count Social Choice and Welfare, Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 Shared Tasks on Quality Estimation. In Proceedings of the Fourth Conference on Machine Translation (WMT). find our card
  6. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  7. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain. In Proceedings of the Sixth Conference on Machine Translation (WMT). Proceedings of the Fourth Conference on Machine Translation (WMT). find our card
  8. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges. In Proceedings of the Fifth Conference on Machine Translation (WMT). find our card
  9. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  10. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b. Results of the WMT20 Metrics Shared Task. In Proceedings of the Fifth Conference on Machine Translation (WMT). find our card
  11. Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  13. Tin Kam Ho, Jonathan J. Hull, and Sargur N. Srihari. 1994. Decision Combination in Multiple Classifier Systems. IEEE transactions on Pattern Analysis and Machine Intelligence, 16(1):66–75. find
  14. Mary Phuong and Christoph Lampert. 2019. Towards Understanding Knowledge Distillation. In Proceedings of the 36th International Conference on Machine Learning (ICML). find
  15. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL). find our card
  16. Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation (WMT). find our card
  17. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An Open Source Framework for Quality Estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo). find
  18. Yoon Kim and Alexander M Rush. 2016. Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  19. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation. In Proceedings of the Seventh Conference on Machine Translation (WMT). find
  20. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open Sesame: Getting inside BERT’s Linguistic Knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (BlackBoxNLP@ACL). find our card
  21. Chi-kiu Lo. 2019. YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources. In Maja Popovic. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation (WMT). find our card
  22. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020a. Intelligent Translation Memory Matching and Retrieval with Sentence Encoders. arXiv preprint arXiv:2004.12894. ↗
  23. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020b. TransQuest: Translation Quality Estimation with Cross-lingual Transformers. In Proceedings of the 28th International Conference on Computational Linguistics (COLING). find our card
  24. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  25. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8:842–866. find our card
  26. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020a. BLEURT: learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  27. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task. In Proceedings of the Fifth Conference on Machine Translation (WMT). find
  28. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers (ATMA). find our card
  29. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020. Findings of the WMT 2020 Shared Task on Quality Estimation. In Proceedings of the Fifth Conference on Machine Translation (WMT). find our card
  30. Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. 2021. Findings of the WMT 2021 Shared Task on Quality Estimation. In Proceedings of the Sixth Conference on Machine Translation (WMT). find our card
  31. Kosuke Takahashi, Katsuhito Sudoh, and Satoshi Nakamura. 2020. Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  32. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL). find our card
  33. Brian Thompson and Matt Post. 2020. Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NIPS). find
  35. Yu Wan, Dayiheng Liu, Baosong Yang, Tianchi Bi, Haibo Zhang, Boxing Chen, Weihua Luo, Derek F. Wong, and Lidia S. Chao. 2021. RoBLEURT Submission for WMT2021 Metrics Task. In Proceedings of the Sixth Conference on Machine Translation (WMT). find
  36. Yu-An Wang and Yun-Nung Chen. 2020. What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  37. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  38. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations (ICLR). find
  39. Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find