Elamur Research
☰
home / papers / 2021 / MuRIL: Multilingual Representations for Indian Languages

MuRIL: Multilingual Representations for Indian Languages

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2103.10730

Summary

Multilingual language models trained on over a hundred languages at once tend to underserve Indian languages. These languages get only a thin slice of the vocabulary and pre-training data, and the models handle romanized or code-mixed Indian text poorly, even though such text is common on social media. The authors present MuRIL, a BERT base encoder pre-trained only on English plus 16 Indian languages. Monolingual text from the Common Crawl OSCAR corpus and Wikipedia trains a Masked Language Modeling objective. A Translation Language Modeling objective uses two kinds of parallel pairs: translations (PMINDIA plus corpora machine-translated into English) and Latin-script versions (Dakshina Dataset plus Wikipedia transliterated with indic-trans). Smaller languages are upsampled, and a dedicated cased WordPiece vocabulary breaks Indian words into fewer pieces than mBERT does. Under zero-shot transfer from English on XTREME tasks, MuRIL beats mBERT on every task for Indian-language test sets, and the gap widens sharply on transliterated test sets. The public release gives Indian-language NLP a stronger base model, including for informal romanized input.

Key points

  • Supported languages: as, bn, gu, hi, kn, ks, ml, mr, ne, or, pa, sa, sd, ta, te, ur, plus English. Wikipedia data is upsampled with a smoothing exponent α = 0.3.
  • Model scale: 236M parameters, about 16B unique tokens, and a vocabulary of 197,285 WordPieces. Training ran for 1M steps (batch 4096, sequence length 512, AdamW, learning rate 5e-4).
  • XTREME (IN) average: 68.6 for MuRIL vs. 59.1 for mBERT. Examples include PANX F1 77.6 vs. 58.0, XNLI accuracy 74.1 vs. 66.8, and TyDiQA-GoldP F1 75.4 vs. 63.1.
  • XTREME (IN-tr), with test sets transliterated to Latin: average 48.9 vs. 21.1. PANX F1 rises from 14.2 to 57.7.
  • Release: the encoder and a pre-processor are on TFHub, and the pre-trained model with its MLM head is on HuggingFace as google/muril-base-cased.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

250 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 246 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2103.10730 · arXiv 2103.10730

@misc{khanuja2021muril,
  title = {{MuRIL: Multilingual Representations for Indian Languages}},
  author = {Khanuja, Simran and Bansal, Diksha and Mehtani, Sarvesh and Khosla, Savya and Dey, Atreyee and Gopalan, Balaji and Margam, Dilip Kumar and Aggarwal, Pooja and Nagipogu, Rajiv Teja and Dave, Shachi and Gupta, Shruti and Gali, Subhash Chandra Bose and Subramanian, Vish and Talukdar, Partha},
  year = {2021},
  doi = {10.48550/arxiv.2103.10730},
  eprint = {2103.10730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.10730}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. find our card
  2. Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tammewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015. Iiit-h system submission for fire2014 shared task on transliterated search. In Proceedings of the Forum for Information Retrieval Evaluation, FIRE ’14, pages 48–53, New York, NY, USA. ACM. find
  3. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546, Online. Association for Computational Linguist… find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  5. Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. Filter: An enhanced fusion method for cross-lingual language understanding. arXiv preprint arXiv:2009.05166. — [arXiv:2009.05166](https://arxiv.org/abs/2009.05166) find
  6. Barry Haddow and Faheem Kirefu. 2020. Pmindia–a collection of parallel corpora of languages of india. arXiv preprint arXiv:2001.09907. — [arXiv:2001.09907](https://arxiv.org/abs/2001.09907) find
  7. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080. find our card
  8. Census of india, 2011. https: //www.censusindia.gov.in/2011Census/C-16\_25062018\_NEW.pdf . find
  9. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  10. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Ass… find our card
  11. Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volum… find
  12. Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirsahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: the dakshina dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2413–2423, Marsei… find
  13. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. find
  14. Statista. 2020. Statista, 2020. https: //www.statista.com/statistics/255146/number-of-internet-users-in-india/ . find
  15. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. find
  16. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics. find our card
  17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find