Elamur Research⌕
☰
home / papers / 2020 / Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

Paper · 2020 · Aditya Siddhant, Ankur Bapna, Yuan Cao et al.

Summary

Multilingual NMT helps low-resource languages by transferring knowledge from high-resource ones, but it usually ignores monolingual text. Self-supervised pre-training, in contrast, uses monolingual text but not multilingual parallel data. Siddhant et al. combine the two. They train a single Transformer on parallel data with a translation objective and, at the same time, on monolingual data with an adapted MASS denoising objective. The data covers 15 WMT languages translated to and from English.

Adding monolingual data improves every language pair, with the largest gains for very low-resource languages such as Kazakh, Turkish and Gujarati. It also improves zero-shot translation between non-English pairs, bringing it close to pivot-based systems without any alignment or adversarial losses. When a language's parallel data is removed entirely, monolingual data alone still produces strong translation quality for that language. This makes it practical to add new languages to existing multilingual models without parallel corpora or back-translation.

Key points

  • Data: WMT parallel data for 30 English-centric pairs, ranging from over 60M sentence pairs (En-Cs) to about 10k (En-Gu). Extra monolingual text for 16 languages comes from news-crawl, common-crawl, europarl-v9, wikidump and other sources, ranging from 2M sentences (Zh) to 270M (De).
  • Setup:
    • Transformer Big (375M parameters) in Tensorflow-Lingvo.
    • Shared 64k SentencePiece vocabulary.
    • Target-language <2xx> token.
    • Temperature sampling with T = 5 for parallel data.
    • Equal 50/50 mix of parallel batches and MASS (monolingual) batches.
  • Low-resource gains: For the lowest-resource languages, monolingual data adds roughly 3–5 BLEU for en→xx and more than 5 BLEU for xx→en over the supervised multilingual model.
  • Zero-shot: Zero-shot quality rises by about 3–6 BLEU. For example, fr→de in the 30-language model improves from 11.8 to 18.5 BLEU, against 21.9 for pivoting.
  • New languages: Using only monolingual data for Romanian gives 33.0 BLEU on WMT ro→en, versus 33.2 with full supervision. This beats XLM by more than 1.5 BLEU and matches bilingual MASS, with no back-translation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 958 worksMultilingual & cross-lingual NLP · 467 works

Cite

Cite as Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan and Yonghui Wu. 2020. Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.252

@inproceedings{siddhant2020leveraging,
  title = {{Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation}},
  author = {Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan and Yonghui Wu},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.252},
  url = {https://doi.org/10.18653/v1/2020.acl-main.252}
}

References

28 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. arXiv preprint arXiv:1903.07091. — [arXiv:1903.07091](https://arxiv.org/abs/1903.07091) find
  3. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019b. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
  4. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations.
  5. Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. 2010. Semi-Supervised Learning, 1st edition. The MIT Press. find
  6. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Association for Computational Linguistics.
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  8. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems. find
  9. Anna Currey and Kenneth Heafield. 2019. Zero-resource neural machine translation with monolingual pivot data. In Workshop on Neural Generation and Translation. find
  10. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  12. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  13. Thanh-Le Ha, Jan Niehues, and Alex Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In International Workshop on Spoken Language Translation.
  14. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Workshop on Neural Machine Translation and Generation.
  15. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems. find
  16. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational… find
  17. Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2019. Pivot-based transfer learning for neural machine translation between non-english languages. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing. find
  18. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Empirical Methods in Natural Language Processing: System Demonstrations.
  19. Surafel M Lakew, Alina Karakanta, Marcello Federico, Matteo Negri, and Marco Turchi. 2019. Adapting multilingual neural machine translation to unseen languages. In International Workshop on Spoken Language Translation. find
  20. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
  21. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Empirical Methods in Natural Language Processing.
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Association for Computational Linguistics.
  23. Matt Post. 2018. A call for clarity in reporting bleu scores. In Conference on Machine Translation.
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  25. Jonathan Shen, Patrick Nguyen, Yonghui Wu, and Zhifeng Chen et al. 2019. Lingvo: a modular and scalable framework for sequence-to-sequence modeling. arXiv preprint arXiv:1902.08295. — [arXiv:1902.08295](https://arxiv.org/abs/1902.08295) find
  26. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning.
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
  28. Ivan Vulić, Goran Glavaš, Roi Reichart, and Anna Korhonen. 2019. Do we really need fully unsupervised cross-lingual embeddings? In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siddhant-2020-leveraging-monolingual-data-self-supervision/