Elamur Research⌕
☰
home / papers / 2026 / MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language

MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language

Paper · 2026 · Sumit Yadav, Raju Kumar Yadav, Utsav Maskey et al.

Summary

Maithili, an Indo-Aryan language spoken by millions in Nepal's Terai and in Bihar, has almost no pretrained models or standard datasets. Models built for Hindi or Nepali transfer imperfectly because Maithili has its own vocabulary and morphology. The authors assemble a Maithili pretraining corpus of roughly 18 million words (over 450MB after cleaning) from Wikipedia, OCR-digitised books and online news outlets. They use it to train maiBERT, a BERT-base-sized encoder (about 110M parameters) trained with a masked language modelling objective and a 30,000-token SentencePiece-BPE vocabulary. For evaluation, they build a ten-class Maithili news classification dataset labelled by native speakers and linguists. Fine-tuned maiBERT reaches 87.02% accuracy. This is only slightly above HindiBERT (86.89%), but maiBERT also beats NepBERTa, mBERT and MuRIL, and the authors report 5–7% gains on categories where Maithili-specific language matters most. Classical machine-learning baselines trail by about ten points. The model and data are released on Hugging Face as a foundation for further Maithili NLP work, such as NER or sentiment analysis.

Key points

  • Pretraining corpus: about 18M words in 1,028,017 sentences, with 1,357,081 unique tokens. Sources are Maithili Wikipedia (~880K words), 418 scanned books processed with Tesseract OCR (~10.4M) and news sites such as Esamaad and Maithilijindabad (~6.6M).
  • Model setup: TensorFlow BERT with 768-dimensional embeddings and sequences of up to 512 tokens. Training ran for 1,000,000 steps with batch size 64 and learning rate 5×10⁻⁵. The paper reports 4.6 GB memory use and 0.85 hours of compute time.
  • Classification dataset: ten news categories, with Culture, Literature and Politics the largest. It was annotated by 3 native speakers and 2 experts (Cohen's Kappa 0.82) and split 70/15/15. The text gives 11,959 instances, but Table 1 totals 12,383.
  • Results: maiBERT scores 87.02% versus HindiBERT's 86.89%, a gap of only 0.13 points. Among classical baselines, the best is linear SVM at 76.70%, and a simple neural network reaches 72.62%.
  • Resources: the model is available as rockerritesh/maiBERT_TF and the data as rockerritesh/maithiliNewsData on Hugging Face.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Macquarie UniversitySydney · Australia42 works · 5,776 citations🎓Institute of EngineeringPulchowk · Nepal2 works · 2 citations🎓Pulchowk CampusPatan · Nepal2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksLow-resource languages · 267 works

Cite

Cite as Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Ganesh Gautam and Usman Naseem. 2026. MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.38

@inproceedings{yadav2026maibert,
  title = {{MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language}},
  author = {Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Ganesh Gautam and Usman Naseem},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.38},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.38}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Raza Ali, Umar Farooq, Umair Arshad, Waseem Shahzad, and Mirza Omer Beg. 2022. Hate speech detection on twitter using transfer learning. Computer Speech & Language, 74:101365. find
  2. Marwah Alian and Arafat Awajan. 2020. Semantic similarity for english and arabic texts: a review. Journal of Information & Knowledge Management, 19(04):2050033. find
  3. Niyati Bafna and Zdeněk Žabokrtskỳ. 2022. Subword-based cross-lingual transfer of embeddings from hindi to marathi and nepali. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 61–71. find
  4. Bal Krishna Bal, Balaram Prasain, Rupak Raj Ghimire, and Praveen Acharya. 2024. Strategies for corpus development for low-resource languages: Insights from nepal. Automatic Speech Recognition and Translation for Low Resource Languages, pages 297–330. find
  5. Dhivya Chandrasekaran and Vijay Mago. 2021. Evolution of semantic similarity—a survey. Acm Computing Surveys (Csur), 54(2):1–37. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  7. Hila Gonen, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. 2020. It’s not greek to mbert: inducing word-level translations from multilingual bert. arXiv preprint arXiv:2010.08275. find
  8. Nathan Hartmann, Erick Fonseca, Christopher Shulby, Marcos Treviso, Jessica Rodrigues, and Sandra Aluisio. 2017. Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv:1708.06025. find
  9. Raviraj Joshi. 2022. L3cube-hindbert and devbert: Pre-trained bert transformer models for devanagari based hindi and marathi languages. arXiv preprint arXiv:2211.11418. — [arXiv:2211.11418](https://arxiv.org/abs/2211.11418) find
  10. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, and 1 others. 2021. Muril: Multilingual representations for indian languages. arXiv preprint arXiv:2103.10730.
  11. Candy Lalrempuii, Badal Soni, and Partha Pakray. 2021. An improved english-to-mizo neural machine translation. Transactions on Asian and Low-Resource Language Information Processing, 20(4):1–21. find
  12. Utsav Maskey, Manish Bhatta, Shiva Bhatt, Sanket Dhungel, and Bal Krishna Bal. 2022. Nepali encoder transformers: An analysis of auto encoding transformer language models for Nepali text classification. In Proceedings of the 1st Annual Meeting of the EL-RA/ISCA Special Interest Group on Under-Resour… find
  13. Rajesh Kumar Mundotiya, Swasti Mishra, and Anil Kumar Singh. 2022. Hierarchical self attention based sequential labelling model for bhojpuri, maithili and magahi languages. Journal of King Saud University-Computer and Information Sciences, 34(10):8739– 8749. find
  14. Rajesh Kumar Mundotiya, Manish Kumar Singh, Rahul Kapur, Swasti Mishra, and Anil Kumar Singh. 2021. Linguistic resources for bhojpuri, magahi, and maithili: statistics about them, their similarity estimates, and baselines for three applications. Transactions on Asian and Low-Resource Language Inform… find
  15. Ankur Priyadarshi and Sujan Kumar Saha. 2020. Towards the first maithili part of speech tagger: Resource creation and system development. Computer Speech & Language, 62:101054. find
  16. Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. 2023. Nepalibert: Pre-training of masked language model in nepali corpus. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), pages 325–330. IEEE. find
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. find
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  19. Joshi Ramchandra, Purvi Goel, and Raviraj Joshi. 2019. Deep learning for hindi text classification: a comparison. In International Conference on Intelligent Human Computer Interaction. Springer. find
  20. R. Smith. 2007. An overview of the tesseract ocr engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 629–633. find
  21. Sulav Timilsina, Milan Gautam, and Binod Bhattarai. 2022. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd conference of the Asia-pacific chapter of the association for computational linguistics and the 12th international joint conference on natural language proce… find
  22. Shuang Zhang, Xuefeng Zheng, and Changjun Hu. 2015. A survey of semantic similarity and its application to social network analysis. In 2015 IEEE International Conference on Big Data (Big Data), pages 2362– 2367. IEEE. find
  23. Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich Schütze. 2022. Cross-lingual retrieval augmented prompt for low-resource languages. arXiv preprint arXiv:2212.09651. find
  24. Rajat Pandit, Saptarshi Sengupta, Sudip Kumar Naskar, Niladri Sekhar Dash, and Mohini Mohan Sardar. 2019. Improving semantic similarity with cross-lingual resources: a study in bangla—a low resourced language. In Informatics, volume 6, page 19. MDPI. find
  25. Rupali S Patil and Satish R Kolhe. 2022. Supervised classifiers with tf-idf features for sentiment analysis of marathi tweets. Social Network Analysis and Mining, 12(1):51. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yadav-2026-maibert-pre-training-corpus-language/