Elamur Research⌕
☰
home / papers / 2026 / NE-LID: A Fast and Accurate Language Identification System for Northeast Indian Languages

NE-LID: A Fast and Accurate Language Identification System for Northeast Indian Languages

Paper · 2026 · Badal Nyalang

Summary

Text-based language identification for Northeast India has received little attention. Popular multilingual LID tools either omit the region's languages or misidentify them despite claiming support. The eleven languages NE-LID handles are Meitei, Mizo, Khasi, Garo, Nyishi, Kokborok, Bodo, Nagamese and Assamese, alongside Hindi and English. They span four families and three scripts (Latin, Bengali-Assamese, Devanagari).

The author sampled 2,000 sentences per language from the NE-BERT corpus, which consists mostly of web-sourced news, social media and digitized documents, and split them 70/15/15. A fastText supervised classifier was then trained on character 2–5-grams plus word unigrams.

On the held-out test set, NE-LID reaches 99.09% accuracy, well ahead of GlotLID, OpenLID, IndicLID and LangDetect. Several of these baselines score zero on multiple languages. The model runs on CPU at about 0.084 ms per prediction.

The work matters because reliable LID is the entry point for collecting corpora and building downstream tools for these under-resourced languages. Both the model and the data are openly released on HuggingFace.

Key points

  • Data: 22,000 sentences in total, split into 15,400 train, 3,300 dev and 3,300 test (300 test sentences per language). About 90% of the content comes from public sources.
  • Accuracy: NE-LID scores 99.09% on test and 99.00% on dev. Baselines score far lower: GlotLID 73.12%, OpenLID 42.03%, IndicLID 39.30%, LangDetect 24.33%. NE-LID stays above 95% on every language.
    • GlotLID scores 0% on Garo and Nagamese.
    • OpenLID misses six languages entirely.
    • IndicLID manages only 54.67% on Hindi.
    • The paper's "2.7× improvement" claim does not match these figures; the gap is 25.97 percentage points.
  • Errors: There are 30 misclassifications, some of which stem from label noise in the source corpus. The main confusions are Bodo–Hindi and Khasi–Garo. Short inputs under 50 characters are hardest, at 98.37% accuracy.
  • Speed and model choice: The model handles about 12,000 predictions per second on CPU. The authors report that transformer attempts (NE-BERT, XLM-R) reached only 9–37%, though with few details given.
  • Limitations and resources: Code-mixed text is not handled, and languages such as Ao, Tangkhul and Dimasa are not covered. The model is at huggingface.co/MWirelabs/ne-lid and the corpus at Badnyal/ne-multilingual-corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Badal Nyalang. 2026. NE-LID: A Fast and Accurate Language Identification System for Northeast Indian Languages. Proceedings of the 8th Workshop on Indian Language Data: Resources and Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2q69fiyg73p6

@inproceedings{nyalang2026ne,
  title = {{NE-LID: A Fast and Accurate Language Identification System for Northeast Indian Languages}},
  author = {Badal Nyalang},
  booktitle = {Proceedings of the 8th Workshop on Indian Language Data: Resources and Evaluation},
  year = {2026},
  doi = {10.63317/2q69fiyg73p6},
  url = {https://doi.org/10.63317/2q69fiyg73p6}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865–879, Toronto, Canada. Association for…
  2. Chuya China Bhanja, Mohammad Azharud-din Laskar, and Rabul Hussain Laskar. 2019. A pre-classification-based language identification for northeast indian languages using prosody and spectral features. Circuits Syst. Signal Process., 38(5):2266–2296. find
  3. Yash Ingle and Pruthwik Mishra. 2025. ILID: Native script language identification for Indian languages. arXiv preprint arXiv:2507.11832. — [arXiv:2507.11832](https://arxiv.org/abs/2507.11832) find
  4. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  5. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12028–12051, Singapore. Association for Computational Linguistics.
  6. Yash Madhani, Mitesh M. Khapra, and Anoop Kunchukuttan. 2023. Bhasa-Abhijnaanam: Native-script and romanized language identification for 22 Indic languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 816–826, Toronto, Ca… find
  7. Shuyo Nakatani. 2010. Language detection library for java. find
  8. Badal Nyalang. 2026. NE-BERT: A multilingual language model for nine Northeast Indian languages. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 1–12, Rabat, Morocco. Association for Computational Linguistics. find
  9. Viyazonuo Terhiija, Samudra Vijaya, and Priyankoo Sarmah. 2019. Spoken language technology for north-east indian languages. In Proceedings of the 1st International Conference on Language Technologies for All, pages 182–185, Paris, France. European Language Resources Association (ELRA). find
  10. Atnafu Lambebo Tonja, Hellina Hailu Nigatu, Olga Kolesnikova, Grigori Sidorov, and Alexander Gelbukh. 2023. First attempt at building parallel corpora for machine translation of Northeast India’s very low-resource languages. arXiv preprint arXiv:2312.04764. 11. Language Resource References [1] MWire… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nyalang-2026-ne-lid-fast-accurate-language/