Elamur Research
☰
home / papers / 2020 / TransQuest: Translation Quality Estimation with Cross-lingual Transformers

TransQuest: Translation Quality Estimation with Cross-lingual Transformers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.coling-main.445.pdf

Summary

Sentence-level quality estimation (QE) predicts how good a machine translation is without a reference translation. Earlier neural QE systems, such as Predictor-Estimator models in deepQuest and OpenKiwi, need large parallel corpora for pre-training. They are costly to train and usually have to be retrained for every language pair. The authors present TransQuest, an open-source PyTorch framework that puts a lightweight output layer on top of the pre-trained cross-lingual XLM-R-large model. It does not use any parallel data. The framework includes two architectures: MTransQuest and the siamese STransQuest. The authors evaluated both on WMT QE datasets covering 15 language pairs. The models outperform existing open-source QE tools and set new state-of-the-art results. A single model can also serve several language pairs at once. For low-resource pairs, initialising from models trained on mid-resource data gives strong results with very few annotated examples. This makes practical QE affordable for many languages, and the released pre-trained model zoo supports this.

Key points

  • Both architectures use XLM-R-large from HuggingFace Transformers, which covers 104 languages. MTransQuest is consistently more accurate, while STransQuest is faster. On a Tesla K80, training on 7,000 instances takes about 4,480s for MTransQuest and 3,900s for STransQuest. Inference on 1,000 instances takes 35s and 16s respectively.
  • A tuned TransQuest won all 8 tasks of the WMT 2020 sentence-level direct assessment (DA) QE shared task.
  • A model for one language pair takes about 2GB. A multi-language-pair model stays at or below 2.1GB, so one model can replace several.
  • Transfer learning started from TransQuest-Mid, a model trained on Et-En, Ro-En and Ru-En. For Ne-En with only 100 training pairs, MTransQuest reached a Pearson correlation of 0.7417, compared with 0.1242 when trained from scratch. With 7,000 pairs it reached 0.7914. Si-En showed similar gains.
  • The code and the TransQuest model zoo of pre-trained models are available at github.com/tharindudr/transquest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

88 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.445

@inproceedings{ranasinghe2020transquest,
  title = {{TransQuest: Translation Quality Estimation with Cross-lingual Transformers}},
  author = {Ranasinghe, Tharindu and Orasan, Constantin and Mitkov, Ruslan},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.445},
  url = {https://doi.org/10.18653/v1/2020.coling-main.445}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32, pages 7059–7069. Curran Associates, Inc. find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July. Association for Computational Linguistics. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  4. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, Vol 8 (2020). find our card
  5. Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1–10, Florence, Italy, August. Association for Computational Linguistics. 3 Several workshops like EM C 2 (Workshop on Energy Efficient Machine Learning and Cognitive Computing), SustaiNLP 2020 (Workshop on Simple and Efficient Natural Language Processing) has been organised on this aspect. find our card
  6. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191, Denver, Colorado, May– June. Association for Computational Linguistics. find
  7. Yvette Graham, Timothy Baldwin, Meghan Dowling, Maria Eskevich, Teresa Lynn, and Lamia Tounsi. 2016. Is all that glitters in machine translation quality estimation really gold? In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3124–3134, Osaka, Japan, December. find
  8. The COLING 2016 Organizing Committee. find
  9. Yvette Graham, Timothu Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  10. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. deepQuest: A framework for neural-based quality estimation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3146–3157, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. find
  11. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. find
  12. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy, July. Association for Computational Linguistics. find
  13. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 562–568, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  14. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality estimation from ScraTCH (QUETCH): Deep learning for word-level translation quality estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisbon, Portugal, September. Association for Computational Linguistics. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  16. André F. T. Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel’s participation in the WMT16 word-level translation quality estimation shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 806–811, Berlin, Germany, August. Association for Computational Linguistics. find
  17. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota, June. Association for Computational Linguistics. find our card
  18. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy, July. Association for Computational Linguistics. find our card
  19. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2019. Semantic textual similarity with Siamese neural networks. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1004–1011, Varna, Bulgaria, September. INCOMA Ltd. find
  20. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest at WMT2020: Sentence-Level Direct Assessment. In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers, online, November. Association for Computational Linguistics. find
  21. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, November. Association for Computational Linguistics. find
  22. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. J. Artif. Int. Res., 65(1):569–630, May. find
  23. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EM C 2 Workshop. find
  24. Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. QuEst - a translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 79–84, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  25. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with QuEst++. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China, July. Association for Computational Linguistics and The Asian Federation of Natural Language Processing. find
  26. Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón Astudillo, and André F. T. Martins. 2018. Findings of the WMT 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 689–709, Belgium, Brussels, October. Association for Computational Linguistics. find our card
  27. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André FT Martins. 2020. Findings of the wmt 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers. find our card
  28. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Maosong Sun, Xuanjing Huang, Heng Ji, Zhiyuan Liu, and Yang Liu, editors, Chinese Computational Linguistics, pages 194–206, Cham. Springer International Publishing. find our card
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card