Elamur Research⌕
☰
home / papers / 2026 / Beyond Multilinguality: Typological Limitations in Multilingual Models for Meitei Language

Beyond Multilinguality: Typological Limitations in Multilingual Models for Meitei Language

Summary

The paper asks whether broad multilingual encoders adequately serve Meitei (Manipuri), a low-resource Tibeto-Burman language of Northeast India. Meitei is agglutinative with SOV order, a profile far from the SVO, lightly inflected languages that dominate such models. The author builds MeiteiRoBERTa, a RoBERTa-base encoder trained from scratch with masked language modeling. It uses a dedicated BPE tokenizer and about 76 million words of Bengali-script Meitei, drawn from IndicCorp v2 plus newly crawled news, government, literary and web text.

Against mBERT and MuRIL, the monolingual model reaches much lower perplexity. Its [CLS] embeddings also separate related from unrelated sentence pairs far better, although MuRIL splits words into fewer pieces. The author concludes that efficient tokenization alone does not secure good representations, and that dedicated capacity matters for typologically distant languages.

The evaluation is very small (10 sentences for fertility, 4 sentence pairs for similarity), so the representation results are indicative only. Even so, the release of the model, code and corpus gives Meitei a basic open NLP resource.

Key points

  • Model and resources: MeiteiRoBERTa has 12 layers, 12 heads, hidden size 768 and 125M parameters, with a 52K BPE vocabulary. Training ran for 3 epochs with 15% masking, batch size 256 and LR 6×10⁻⁴ on NVIDIA A40 GPUs. The model is released as MWirelabs/meitei-roberta and the corpus as MWirelabs/meitei-monolingual-corpus on Hugging Face.
  • Perplexity (19,038 validation samples): MeiteiRoBERTa scores 65.89, against 341.56 for mBERT and 355.65 for MuRIL. That is roughly 5.2× and 5.4× better. These are MLM-based scores computed over different tokenizers, so the comparison is not strictly like-for-like.
  • Tokenization fertility (lower is better): MuRIL scores 3.29 with a 197K vocabulary. MeiteiRoBERTa scores 4.65 with 52K, and mBERT 4.79 with 119K.
  • Semantic separation (cosine on [CLS]): MeiteiRoBERTa scores 0.769, compared with 0.035 for mBERT and 0.000 for MuRIL. The two baselines assign near-identical high similarity to every pair.
  • Stated limitations: The model covers Bengali script only, with no Meitei Mayek. The corpus leans toward formal news and government text. There was no community consultation and no bias audit. Downstream tasks such as NER, sentiment analysis and MT are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigtyp-main.5

@inproceedings{nyalang2026beyond,
  title = {{Beyond Multilinguality: Typological Limitations in Multilingual Models for Meitei Language}},
  author = {Nyalang, Badal},
  booktitle = {Proceedings of the 8th Workshop on Research in Computational Linguistic Typology and Multilingual NLP},
  year = {2026},
  doi = {10.18653/v1/2026.sigtyp-main.5},
  url = {https://doi.org/10.18653/v1/2026.sigtyp-main.5}
}

References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for basque. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC), pages 4781–4788. find
  2. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6282–6293. find our card
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), pages 9–15. find our card
  4. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computational… find our card
  5. Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A call for more rigor in unsupervised cross-lingual learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7375–7388. find
  6. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. MuRIL: Multilingual representations for Indian langua… find our card
  7. Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, and Rifat Shahriyar. 2022. BanglaBERT: Language model pretraining and benchmarks for low-resource language understanding evaluation in Bangla. In Findings of the Association for Comp… find our card
  8. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR). find
  9. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR). find
  10. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In Proceedings of the 2020 find our card
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  13. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: A tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), p… find our card
  14. Christopher Moseley. 2010. Atlas of the World’s Languages in Danger, 3rd edition. UNESCO Publishing. find
  15. Abhijnan Nath, Sheikh Mannan, and Nikhil Krishnaswamy. 2023. AxomiyaBERTa: A phonologically-aware transformer model for Assamese. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11629–11646, Toronto, Canada. Association for Computational Linguistics. find
  16. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042. find our card
  17. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673. find our card
  18. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376. find our card
  19. Thoudam Doren Singh and Sivaji Bandyopadhyay. 2009. Named entity recognition for Manipuri using support vector machine. In Proceedings of the 23rd Pacific Asia Conference on Language, Information and Computation (PACLIC), pages 811–818. find
  20. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly universal dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302–2315. find
  21. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nyalang-2026-beyond-multilinguality-typological-limitations-multilingual/