Elamur Research⌕
☰
home / papers / 2021 / Larger-Scale Transformers for Multilingual Masked Language Modeling

Larger-Scale Transformers for Multilingual Masked Language Modeling

Summary

Multilingual masked language models like XLM-R help low-resource languages. However, splitting limited capacity across many languages tends to hurt results on high-resource ones. This paper tests whether a much larger model removes that trade-off.

The authors keep the XLM-R recipe:

  • MLM training on monolingual text only
  • a SentencePiece unigram vocabulary of 250K
  • language sampling with α = 0.3
  • the CC100 corpus (167B tokens, 100 languages)

With this setup they train two new models, XLM-R XL (3.5B parameters) and XLM-R XXL (10.7B), using tensor model parallelism and pre-LayerNorm. Evaluation covers XNLI, MLQA, XQuad and English GLUE.

Both models clearly beat XLM-R Large on cross-lingual tasks and set new best scores for several individual languages. They are competitive with mT5 despite far less pretraining data. XXL even slightly surpasses the English-only RoBERTa-Large on GLUE. The takeaway is that enough capacity lets a single multilingual model serve high- and low-resource languages alike. Code and models are publicly released.

Key points

  • XNLI average accuracy:
    • Zero-shot transfer from English: 80.9 (Large), 82.3 (XL), 83.1 (XXL).
    • Translate-train-all: 83.6, 85.4 and 86.0 for the same models.
    • New state of the art for French, Vietnamese and Hindi.
  • Zero-shot QA (Large → XXL):
    • MLQA gains +4.1 F1 / +3.9 EM.
    • XQuad gains roughly +4.5 F1 / +5.5 EM, with top results for Arabic, German, Greek and Russian.
  • GLUE dev average (MNLI, QNLI, QQP, SST, MRPC): XLM-R XXL scores 93.2, RoBERTa-Large 92.9 and XLM-R Large 91.9.
  • Setup:
    • XL: L=36, H=2560. XXL: L=48, H=4096. Both use A=32.
    • Batch size 2048, 500K updates (~0.5T tokens seen), sequence length 512.
    • Model-parallel size is 2 for XL and 8 for XXL.
  • Comparison with mT5:
    • mT5 is trained on mC4 (6.4T tokens, ~38× CC100).
    • XLM-R XL beats mT5-XL by 0.6 on XNLI translate-train-all, 1.3 EM on XQuad and 0.9 EM on MLQA.
    • mT5-XXL leads XLM-R XXL by 1.4 on average XNLI zero-shot and by 8.4 on Swahili.
    • The authors suggest CC100 may be getting saturated at this scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.repl4nlp-1.4

@inproceedings{goyal2021larger,
  title = {{Larger-Scale Transformers for Multilingual Masked Language Modeling}},
  author = {Goyal, Naman and Du, Jingfei and Ott, Myle and Anantharaman, Giri and Conneau, Alexis},
  booktitle = {Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)},
  year = {2021},
  doi = {10.18653/v1/2021.repl4nlp-1.4},
  url = {https://doi.org/10.18653/v1/2021.repl4nlp-1.4}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019. find our card
  3. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. ACL. find our card
  4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856. find our card
  5. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. find
  6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Proc. of NeurIPS. find
  7. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. find
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  10. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In EMNLP. Association for Computational Linguistics. find
  11. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. find
  13. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359. find our card
  14. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. find
  15. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2nd Workshop on Evaluating Vector-Space Representations for NLP. find
  16. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL, pages 66–75. find
  17. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. find
  18. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP. find our card
  19. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. NeurIPS. find
  20. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. find our card
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/