Elamur Research⌕
☰
home / papers / 2026 / MUTANT: A Recipe for Multilingual Tokenizer Design

MUTANT: A Recipe for Multilingual Tokenizer Design

Paper · 2026 · Souvik Rana, Ashish Kulkarni, Arul Menezes et al.

Summary

Tokenizers built mainly for English break Indian-language text into many small pieces. The LLaMA-4 tokenizer, for instance, needs roughly 10.5 tokens per word for Oriya. This raises training and inference costs and treats languages unequally. The authors propose MUTANT, a reproducible procedure for designing multilingual tokenizers, and use it to build MUTANT-Indic. It covers English, the 22 scheduled languages of India, and code. The recipe has three parts:

  • It filters and normalizes a varied corpus drawn from web text, Wikipedia, math and code.
  • It sets per-script vocabulary budgets by adjusting the proportions of the training data, rather than gluing separately trained tokenizers together.
  • It trains byte-level BPE in two phases. The first allows merges only inside words; the second permits merges across word boundaries to capture multiword expressions.

MUTANT-Indic cuts average fertility by 39.5% compared with LLaMA4 and by 18% compared with Sutra, the strongest prior tokenizer. It also gives 44% higher inference throughput than LLaMA4 while keeping English and Indic benchmark results comparable. The work shows that careful tokenizer design yields cheaper and more equitable multilingual LLMs.

Key points

  • Evaluation scope: 22 Indic languages, English and code. The authors released an evaluation framework (github.com/ola-krutrim/MUTANT) and a dataset (krutrim-ai-labs/MUTANT on Hugging Face).
  • Two-stage training: The curriculum follows the SuperBPE approach. The best switch from subword to cross-word merges comes at 85–90% of the target vocabulary. This two-stage design beat the single-stage BoundlessBPE on fertility for the top 10 Indic languages and English.
  • Vocabulary allocation: Matching the training corpus to the desired per-script budgets worked better than explicitly merging script-specific vocabularies. The merged approach caused interference and inconsistent segmentation across scripts.
  • Ablation findings:
    • Gains plateau beyond about 10 GB of Stage 1 training data.
    • Vocabularies larger than 200K bring no notable fertility gain.
    • NFKC normalization gives small but consistent improvements.
  • Limitations:
    • Downstream tests use models of at most 1B parameters, in both pretraining-from-scratch and continual-pretraining settings.
    • Morphology-aware pre-tokenization was studied but left out of the final tokenizer because of its high inference latency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksTokenization & segmentation · 124 works

Cite

Cite as Souvik Rana, Ashish Kulkarni, Arul Menezes, Chandra Khatri and Shubham Agarwal. 2026. MUTANT: A Recipe for Multilingual Tokenizer Design. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2146

@inproceedings{rana2026mutant,
  title = {{MUTANT: A Recipe for Multilingual Tokenizer Design}},
  author = {Souvik Rana, Ashish Kulkarni, Arul Menezes, Chandra Khatri and Shubham Agarwal},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2146},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2146}
}

References

63 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, and 1 others. 2025. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318.
  2. ADA Sci. 2024. Multilingual tokenization efficiency in large language models: A study on Indian languages. Lattice, 5(2). find
  3. AI4Bharat. 2022. IndicBERT: A multilingual AL-BERT model for Indian languages. https://huggingface.co/ai4bharat/indic-bert. ↗
  4. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, and 1 others. 2024. Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 202…
  5. Maharaj Brahma, NJ Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Morphtok: Morphologically grounded tokenization for indian languages. arXiv preprint arXiv:2504.10335. — [arXiv:2504.10335](https://arxiv.org/abs/2504.10335) find
  6. Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, and 1 others. 2024. Towards effective and efficient continual pre-training of large language models. arXiv preprint arXiv:2407.18743.
  7. Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, and Ivan P. Yamshchikov. 2024. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604. Association for Computational… find
  8. Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv preprint arXiv:2402.01035.
  9. Anh Dang, Limor Raviv, and Lukas Galke. 2024. Tokenization and morphology in multilingual language models: A comparative analysis of mT5 and ByT5. arXiv preprint arXiv:2410.11627. find
  10. DeepSeek. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. https://github.com/deepseek-ai/DeepSeek-R1. ↗
  11. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. Preprint, arXiv:2212.05409.
  12. Abhimanyu Dubey and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  14. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  15. Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. 2024. Coercing llms to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020. — [arXiv:2402.14020](https://arxiv.org/abs/2402.14020) find
  16. Gemma Team. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  18. Shuhao Gu, Mengdi Zhao, Bowen Zhang, Liangdong Wang, Jijie Li, and Guang Liu. 2024. Retok: Replacing tokenizer to enhance representation efficiency in large language model. Preprint, arXiv:2410.04335. — [arXiv:2410.04335](https://arxiv.org/abs/2410.04335) find
  19. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  20. Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference attack: Bpe tokenizers reveal training data compositions. Advances in Neural Information Processing Systems, 37:8956–8983. find
  21. David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A Smith, Hannaneh Hajishirzi, Kyle Lo, and Jesse Dodge. 2025. Signal and noise: A framework for reducing uncertainty in language model evaluation. arXiv preprint arXiv:2508.13144. — [arXiv:2508.13144](https://arxiv.org/abs/2508.13144) find
  22. Shashank Mohan Jain. 2022. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pages 51–67. Springer. find
  23. Divyanshu Kakwani and 1 others. 2020. Monolingual corpora, evaluation benchmarks and pre-trained models for 11 major Indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961. find
  24. N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, and Maunendra Sankar Desarkar. 2025a. Multilingual tokenization through the lens of indian languages: Challenges and insights. arXiv preprint arXiv:2506.17789. find
  25. N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, and Maunendra Sankar Desarkar. 2025b. Multilingual tokenization through the lens of indian languages: Challenges and insights. Preprint, arXiv:2506.17789. find
  26. Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. 2024. Indicllmsuite: A blueprint for creating pre-training and fine-tuning dataset…
  27. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and 1 others. 2022. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research. find
  28. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational…
  29. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
  30. Anoop Kunchukuttan. 2020. The IndicNLP Library. https://github.com/anoopkunchukuttan/ indic_nlp_library/blob/master/docs/ indicnlp.pdf. ↗
  31. Sander Land and Max Bartolo. 2024. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. Preprint, arXiv:2405.05417.
  32. Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Peng Liu, Jungong Han, and Guiguang Ding. 2024. Scaffold-BPE: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal. arXiv preprint arXiv:2404.17808. — [arXiv:2404.17808](https://arxiv.org/abs/2404.17808) find
  33. Qwen. 2024. Qwen3 technical report. qwenlm.github.io/blog/qwen3/. find
  34. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025a. Superbpe: Space travel for language models. Preprint, arXiv:2503.13423. — [arXiv:2503.13423](https://arxiv.org/abs/2503.13423) find
  35. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  36. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025b. SuperBPE: Space travel for language models. arXiv preprint arXiv:2503.13423. — [arXiv:2503.13423](https://arxiv.org/abs/2503.13423) find
  37. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173. find
  38. Daniil Maksymenko and Oleksii Turuta. 2025. Tokenization efficiency of current foundational large language models for the Ukrainian language. PMC Digital Health, 12380774. find
  39. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. ↗
  40. Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, and 1 others. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprint arXiv:2112.10508.
  41. Mistral. 2024. Mistral nemo. https://mistral.ai/ news/mistral-nemo. ↗
  42. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025. 2 ol… find
  43. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  44. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint arXiv:2406.17557.
  45. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963– 36990. https://
  46. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. TODO. find
  47. Varshini Reddy, Craig W Schmidt, Yuval Pinter, and Chris Tanner. 2025. How much is enough? the diminishing returns of tokenization training data. arXiv preprint arXiv:2502.20273. — [arXiv:2502.20273](https://arxiv.org/abs/2502.20273) find
  48. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  49. Sarvam. 2024. Sarvam-1: A 2b parameter language model optimized for indian languages. https://www.sarvam.ai/blogs/sarvam-1. ↗
  50. Sarvam AI. 2025. Arc-challenge-indic. https://huggingface.co/datasets/sarvamai/ arc-challenge-indic. ↗
  51. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, and 1 others. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  52. Craig W. Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter. 2024. Boundless byte pair encoding: Breaking the pre-tokenization barrier. arXiv preprint arXiv:2504.00178. — [arXiv:2504.00178](https://arxiv.org/abs/2504.00178) find
  53. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Li…
  54. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
  55. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. find
  56. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103. Association for Computational Linguistics. find
  57. Sagar Tamang and Dibya Jyoti Bora. 2024. Evaluating tokenizer performance of large language models across official indian languages. arXiv preprint arXiv:2411.12240. — [arXiv:2411.12240](https://arxiv.org/abs/2411.12240) find
  58. Hugo Touvron and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  59. Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. 2024. Milu: A multi-task indic language understanding benchmark. arXiv preprint arXiv: 2411.02538. find
  60. Wenhao Xue and 1 others. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv preprint arXiv:2402.01035.
  61. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  62. Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P Xing. 2025. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.02807. find
  63. Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rana-2026-mutant-recipe-multilingual-tokenizer-design/