Elamur Research⌕
☰
home / papers / 2026 / Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses

Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses

Paper · 2026 · Manikandan Ravikiran, Tanmay Tiwari, Vibhu Gupta et al.

Summary

The paper asks whether standard subword tokenizers break apart informal Hindi expressions (slang, idioms, other lexicalized multi-word phrases) whose meaning holds only as a whole. The authors treat these categories together because they share one problem for segmentation: semantic units do not match word boundaries. Using the HiSlang-4.9k dataset, they compare five tokenizers (multilingual, Indic-focused and byte-level) in three ways:

  • Static metrics for boundary preservation and phrase integrity.
  • Downstream expression identification, framed as BIO sequence labeling under identical training conditions.
  • Robustness tests with orthographic variants, romanization and spelling noise.

On clean Devanagari text, Indic-oriented models (MuRIL, IndicBERT) keep expression boundaries better and reach higher span-level F1 than mBERT and XLM-R. Better static preservation tends to go with better downstream results. Under romanization, however, every tokenizer collapses, with phrase integrity close to zero. The work shows that tokenization is an overlooked bottleneck for informal and cross-script Hindi NLP. It also argues that tokenizer evaluations should cover clean, noisy and romanized input.

Key points

  • Dataset: HiSlang-4.9k contains 4,906 sentences, half with annotated expressions and half without (2,453 each), averaging 15.5 words. The sentences were gathered from film material (screenplays and subtitle files), existing language corpora, and posts on social platforms.
  • Annotation: eight native speakers labeled the data and two linguists reviewed it, with Cohen's κ of 0.97 at sentence level and 0.94 at phrase level. The task uses B-EXP / I-EXP / O tags with an 80/10/10 sentence-level split.
  • Byte-level tokenization: it removes OOV problems but fragments expressions so heavily that downstream performance drops to zero.
  • Noise results: IndicBERT's unusually high span F1 under noise comes from degenerate predictions, not real robustness. This shows that aggregate metrics can hide pathological behavior.
  • Limitations: the study covers Hindi only, and tokenizer, vocabulary and pretraining effects are not separated. The noise is synthetic, and the only task evaluated is sequence labeling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States69 works · 51,069 citations🎓Indian Institute of Technology MandiMandi · India2 works · 3 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Tokenization & segmentation · 124 works

Cite

Cite as Manikandan Ravikiran, Tanmay Tiwari, Vibhu Gupta, Rakesh Prakash, Rohit Saluja and Shayan Mohanty. 2026. Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.2

@inproceedings{ravikiran2026do,
  title = {{Do Tokenizers Fail on Informal Hindi Expressions? Evidence from Static, Downstream, and Robustness Analyses}},
  author = {Manikandan Ravikiran, Tanmay Tiwari, Vibhu Gupta, Rakesh Prakash, Rohit Saluja and Shayan Mohanty},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.2},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.2}
}

References

4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kalika Bali, Jatin Sharma, Monojit Choudhury, and Yogarshi Vyas. 2014. “I am borrowing ya mixing?” an analysis of english-hindi code-mixing in facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116– 126, Doha, Qatar. Association for Computational Ling…
  2. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR). find
  3. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics.
  4. Maharaj Brahma, N. J. Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Morphtok: Morphologically grounded tokenization for indian languages. In Proceedings of the ICML 2025 Workshop on Tokenization (TokShop). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ravikiran-2026-do-tokenizers-fail-informal-hindi/