Elamur Research⌕
☰
home / papers / 2026 / The Impact of Tokenization Algorithms on Hungarian Language Model Performance

The Impact of Tokenization Algorithms on Hungarian Language Model Performance

Summary

Subword tokenization strongly affects how language models handle agglutinative languages, yet no one had systematically tested which algorithm works best for Hungarian. The authors train BPE, WordPiece and Unigram LM tokenizers at four vocabulary sizes (16K to 128K) on a filtered Hungarian portion of OSCAR. They also add a fourth option: a Unigram LM whose seed vocabulary comes from morphemes extracted with the emMorph analyzer rather than from suffix-array statistics. Each tokenizer is assessed with intrinsic quality measures. Each is also used to pretrain a BERT model, which is then fine-tuned on the HuLU benchmark. BPE produces the most compact and morpheme-consistent segmentations. The morphology-seeded Unigram LM achieves the best downstream scores overall. However, most differences between algorithms and vocabulary sizes are not statistically significant, and only HuCoPA responds clearly to both factors. The work gives practical guidance and released code for designing tokenizers for Hungarian and similar morphologically rich languages. It also suggests that compact segmentation alone does not guarantee better task performance.

Key points

  • Data: The Hungarian OSCAR subset (about 18.0M documents, 5.16B words) was reduced by quality filtering to about 6.3M documents and 1.8B words, which were used to train both the tokenizers and the models.
  • Tokenizers: BPE, WordPiece (Hugging Face reconstruction) and Unigram LM, plus a Unigram LM seeded with emMorph morphemes. All were trained with the Hugging Face tokenizers library using whitespace pre-tokenization and NFC normalization. The modified tokenizer's code is at github.com/matee8/tokenizers.
  • Models: BERT-base-sized encoders (12 layers, 12 heads, hidden size 768, sequence length 1024), each pretrained for 2 epochs with batch size 96.
  • Evaluation: Fine-tuning on the five HuLU tasks (HuCOLA, HuCoPA, HuRTE, HuSST, HuCB), followed by Kruskal-Wallis tests with a Bonferroni-corrected threshold of α = 0.00125.
  • Significant effects (HuCoPA only): Vocabulary size mattered for WordPiece (p = 0.0007) and Unigram LM (p = 0.0004). The choice of tokenizer mattered at the 64K vocabulary size (p = 0.0004).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/29hx92kq2dxe

@inproceedings{osvath2026impact,
  title = {{The Impact of Tokenization Algorithms on Hungarian Language Model Performance}},
  author = {Osváth, Mátyás and Molnár, Máté Norbert and Gunics, Roland and Ligeti-Nagy, Noémi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/29hx92kq2dxe},
  url = {https://doi.org/10.63317/29hx92kq2dxe}
}

References

48 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Catherine Arnett, Marisa Hudspeth, and Brendan O’Connor. 2025. Evaluating Morphological Alignment of Tokenizers in 70 Languages. find
  2. Kaj Bostrom and Greg Durrett. 2020. Byte Pair Encoding is Suboptimal for Language Model Pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. find
  3. Iaroslav Chelombitko, Egor Safronov, and Aleksey Komissarov. 2024. Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models. find
  4. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics, 10:73–91. find
  5. Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two Counterexamples to Tokenization and the Noiseless Channel. find
  6. Mathias Creutz, Krista Lagus, Krista Lagus@hut, and Fi. 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. find
  7. Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching Large Language Models New Languages. find
  8. Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Al-Dahle. 2024. The Llama 3 Herd of Models. find
  11. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38… find
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. find
  14. Yanis Labrak, Adrien Bazoge, Beatrice Daille, Mickael Rouvier, and Richard Dufour. 2024. How Important Is Tokenization in French Medical Masked Language Models? find
  15. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942. find
  16. Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. HuLU: Hungarian Language Understanding Benchmark Kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (… find
  17. Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067–1097, Mexico City, Mexico.… find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  19. Beso Mikaberidze, Temo Saghinadze, Guram Mikaberidze, Raphael Kalandadze, Konstantine Pkhakadze, Josef van Genabith, Simon Ostermann, Lonneke van der Plas, and Philipp Müller. 2024. A Comparison of Different Tokenization Methods for the Georgian Language. In Proceedings of the 7th International Conf… find
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. find
  21. Luca Moroni, Giovanni Puccetti, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell’Orletta, Andrea Esuli, and Roberto Navigli. 2025. Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation. find
  22. Attila Novák, Borbála Siklósi, and Csaba Oravecz. 2016. A new integrated open-source morphological analyzer for Hungarian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find
  23. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim… find
  24. Faisal Qarah and Tawfeeq Alsanoosy. 2024. A Comprehensive Analysis of Various Tokenizers for Arabic Large Language Models. Applied Sciences, 14(13):5696. find
  25. Alec Radford, Jeff Wu, R. Child, D. Luan, Dario Amodei, and I. Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. find
  27. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. find our card
  28. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  29. Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization Is More Than Compression. find
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… find
  31. S. Tamang and D. J. Bora. 2024. Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages. find
  32. Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. find
  33. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, and Shreya Pathak. 2024. Gemma: Open Models Based on Gemini Research and Technology. find
  34. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, and Oguzhan Ozcelik. 2023. Impact of Tokenization on Language Models: An Analysis for Turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–21. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. find
  36. Saketh Reddy Vemula, Dipti Misra Sharma, and Parameswari Krishnamurthy. 2025. Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment. find
  37. Veronika Vincze, Katalin Simkó, Zsolt Szántó, and Richárd Farkas. 2017. Universal Dependencies and Morphology for Hungarian - and on the Price of Universality. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages… find
  38. Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Ziqin Luo, Guochao Jiang, Jiaqing Liang, and Deqing Yang. 2025. Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization. find
  39. Anna Wegmann, Dong Nguyen, and David Jurgens. 2025. Tokenization is Sensitive to Language Variation. find
  40. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, and Suzana Ilić. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. find
  41. Shangda Wu, Xu Tan, Zili Wang, Rui Wang, Xiaobing Li, and Maosong Sun. 2024. Beyond Language Models: Byte Models are Digital World Simulators. find
  42. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  43. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291–306. find
  44. Zijian Győző Yang, Ágnes Bánfi, Réka Dodé, Gergő Ferenczi, Péter Hatvani, Enikő Héja, Mariann Lengyel, Gábor Madarasz, Mátyás Osváth, Bence Sárossy, Kristóf Varga, Tamás Váradi, Gábor Prószéky, and Noémi Ligeti-Nagy. 2025a. ChatPULI: Enhancement to the first Hungarian conversational model. Annales M… find
  45. Zijian Győző Yang, Ágnes Bánfi, Réka Dodé, Gergő Ferenczi, Péter Hatvani, Enikő Héja, Mariann Lengyel, Gábor Madarasz, Mátyás Osváth, Bence Sárossy, Kristóf Varga, Tamás Váradi, Gábor Prószéky, and Noémi Ligeti-Nagy. 2025b. PULI Chat Our First Hungarian Conversational Model. Eger, Hungary. Internati… find
  46. Győző Yang Zijian, Enikő Héja, and Tamás Váradi. 2024a. The First Instruct-Following Large Language Models for Hungarian. find
  47. Győző Yang Zijian, Enikő Héja, and Tamás Váradi. 2024b. ParancsPULI: Az utasításkövető PULI-modell. find
  48. Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the Noiseless Channel. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/osvath-2026-impact-tokenization-algorithms-hungarian-language/