Elamur Research⌕
☰
home / papers / 2026 / TamilTok: Morphologically-Informed Tokenization for Tamil

TamilTok: Morphologically-Informed Tokenization for Tamil

Paper · 2026 · Surendhar Muthukumar, Aaricia Herygers and Lisa Beinborn

Summary

Tamil is an agglutinative Dravidian language with sandhi sound changes at morpheme junctions and its own abugida script. It is usually processed with multilingual subword tokenizers. Their frequency-based vocabularies break Tamil words into many fragments that ignore morpheme boundaries.

To address the lack of segmentation data, the authors built TamilMorph, a resource of over 480,000 Tamil word forms annotated with morpheme boundaries. Instead of relying on an unreliable analyzer, they took noun and verb roots from the Tamil Virtual Academy corpus. They then applied inflection and derivation rules, including sandhi, from a Tamil grammar reference to over-generate candidate forms, keeping only those attested in a corpus.

A neural segmenter trained on this data feeds TamilTok, a pipeline that splits words into morphemes before subword vocabulary learning. The authors compared BPE, WordPiece and UnigramLM across several vocabulary sizes. TamilTok achieves better morphological alignment and better named entity recognition than earlier tokenizers, while compression stays competitive. The released data and tokenizer serve as a benchmark and a model for other morphologically rich languages.

Key points

  • TamilMorph construction: 320,448 annotated word forms from the Tamil Virtual Academy corpus were normalized and deduplicated, yielding 28,209 noun roots and 3,078 verb roots. Formation rules come from Chithra et al. (2024), and generated forms are filtered by frequency in the IndicNLP corpus.
  • TamilTok pipeline: a supervised neural morphological segmenter acts as an upstream pre-tokenization step. Tokenizers are evaluated with MorphScore F1 (Morph-F1) and fertility.
  • Main findings:
    • Multilingual tokenizers (e.g., Gemma) heavily over-segment Tamil.
    • Algorithm choice and vocabulary size matter more than the training corpus.
    • UnigramLM beats BPE and WordPiece on Morph-F1, and larger vocabularies improve both compression and alignment.
    • Fertility alone misses qualitative differences between tokenizers.
  • Downstream result: morpheme-aware tokenization improves NER. The dataset and tokenizer are public at gitlab.gwdg.de/huds/projects/tamiltok.
  • Limitations:
    • Some extracted "roots" are actually compounds.
    • The suffix and rule inventory is incomplete.
    • The strict corpus filter may discard valid forms common in domains outside IndicNLP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GöttingenGöttingen · Germany17 works · 158 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Morphology · 405 worksTokenization & segmentation · 119 works

Cite

Cite as Surendhar Muthukumar, Aaricia Herygers and Lisa Beinborn. 2026. TamilTok: Morphologically-Informed Tokenization for Tamil. Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.dravidianlangtech-1.7

@inproceedings{muthukumar2026tamiltok,
  title = {{TamilTok: Morphologically-Informed Tokenization for Tamil}},
  author = {Surendhar Muthukumar, Aaricia Herygers and Lisa Beinborn},
  booktitle = {Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages},
  year = {2026},
  doi = {10.18653/v1/2026.dravidianlangtech-1.7},
  url = {https://doi.org/10.18653/v1/2026.dravidianlangtech-1.7}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Em-Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav…
  2. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. Association for Computational Linguistics. find
  3. Anand Kumar M, Dhanalakshmi V, Soman K.P, and Rajendran S. 2010. A novel data driven algorithm for Tamil morphological generator. International Journal of Computer Applications, 6(12):52–56. find
  4. Mathias Johan Philip Creutz and Bo Krister Johan Linden. 2004. Morpheme segmentation gold standards for Finnish and English. In Publications in Computer and Information Science: Report A77. Helsinki University of Technology. find
  5. Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. AI4Bharat-IndicNLP Corpus: Monolingual corpora and word embeddings for Indic languages. arXiv preprint arXiv:2005.00085. — [arXiv:2005.00085](https://arxiv.org/abs/2005.00085) find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  8. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages. In Proceedings of the 61st Annual Meeting of the Asso…
  9. Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using morphological knowledge in open-vocabulary neural language models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… find
  10. Gemma Team. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  11. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
  12. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  13. Yirong Pan, Xiao Li, Yating Yang, and Rui Dong. 2020. Morphological word segmentation on agglutinative languages for neural machine translation. Preprint, arXiv:2001.01589. — [arXiv:2001.01589](https://arxiv.org/abs/2001.01589) find
  14. Haris Jabbar. 2023. Morphpiece: A linguistic tokenizer for large language models. arXiv preprint arXiv:2307.07262. — [arXiv:2307.07262](https://arxiv.org/abs/2307.07262) find
  15. Iñigo Parra. 2024. Morphological typology in BPE subword productivity and language modeling. In Latinx in AI@ NeurIPS 2024. find
  16. Arjun Sai Krishnan and Seyoon Ragavan. 2021. Morphology-aware meta-embeddings for Tamil. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 94–111, Online. Association for Computational Linguistics. find
  17. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, volume 36, pages 36963–36990. Curran Associates, Inc.
  18. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66– 75, Melbourne, Australia. Association for Computational…
  19. Kengatharaiyer Sarveswaran, Gihan Dias, and Miriam Butt. 2021. Thamizhi Morph: A morphological parser for the Tamil language. Machine Translation, 35(1):37–70. find
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer…
  21. Jothir Adithya T K, Nithish Kumar S, Felicia Lilian J, and Mahalakshmi S. 2024. Monolingual text summarization for Indic languages using LLMs. In Proceedings of the 21st International Conference on Natural Language Processing (ICON), pages 94–101, AU-KBC Research Centre, Chennai, India. NLP Associat… find
  22. Tamil Virtual Academy. 2022. Tamil annotated corpus. https://github.com/Tamil-Virtual-Academy/ Accessed: 2026-Tamil-Annotated-Corpus. 02-25. ↗
  23. Saketh Reddy Vemula, Sandipan Dandapat, Dipti Sharma, and Parameswari Krishnamurthy. 2025. Rethinking tokenization for rich morphology: The dominance of unigram over BPE and morphological alignment. In The 14th International Joint Conference on Natural Language Processing and The 4th Conference of t… find
  24. Sami Virpioja, Peter Smit, Stig-Arne Grönroos, and Mikko Kurimo. 2013. Morfessor 2.0: Python implementation and extensions for morfessor baseline. Aalto University publication series SCIENCE + TECHNOLOGY, 25/2013. find
  25. Wikimedia Foundation. 2023. Wikimedia downloads. Accessed: 10-10-2025. find
  26. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. find
  27. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  28. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/muthukumar-2026-tamiltok-morphologically-informed-tokenization-tamil/