Elamur Research⌕
☰
home / papers / 2026 / TReX: Tokenizer Regression for Optimal Data Mixture

TReX: Tokenizer Regression for Optimal Data Mixture

Paper · 2026 · Inho Won, Hangyeol Yoo, Minkyung Cho et al.

Summary

Multilingual tokenizers compress text unevenly depending on how much data each language contributes. The usual ways to set these language proportions are hand-tuned guesses or expensive repeated full-scale training runs. TReX instead treats mixture selection as a prediction problem. The authors draw many candidate language ratios from a Dirichlet distribution and train cheap, small proxy tokenizers on each one. They score each proxy by Normalized Sequence Length (NSL) and fit a regression model that maps mixture weights to expected compression. That model can then evaluate tens of millions of candidate mixtures almost instantly, and the best-scoring one is used to train the full-size tokenizer. The approach relies on the ranking of mixtures staying stable from small to large corpus and vocabulary scales. On a 19-language setup, the regressor predicts compression closely. The resulting tokenizers compress text up to 12% better than ones built with LLaMA3-style, GPT-4o-style or uniform mixtures, on both in- and out-of-distribution data. This matters because better compression directly cuts LLM training and inference cost.

Key points

  • The regression model reaches a MAPE of 1.989 (under 2%) and a Spearman rank correlation above 0.97. Its predictions remain consistent across settings from 1GB/64k to 30GB/200k (corpus size/vocabulary).
  • Search cost is low:
    • The trained model can score over 50M mixtures within seconds.
    • Building it takes 480 proxy tokenizers.
    • Compared with AdaptMix, which needs about 20 large-scale iterations at 30GB/200K, total training time drops by 52.2%.
  • For a simulated 13B-parameter model trained on 3T tokens, the TReX tokenizer gives the shortest estimated training time (8,009.6 hours). That is about 200 hours less than the GPT-4o-based mixture and over 1,000 hours less than the weakest baseline.
  • In a domain-specific test on the medical portion of the Pile, prediction quality stays high (Spearman >0.965, MAPE 0.921).
  • Stated limitations:
    • Only compression is optimized; effects on downstream task quality are untested.
    • The experiments cover just 19 languages.
    • Rank invariance is unverified at much larger scales or with many morphologically rich and low-resource languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea56 works · 3,269 citations🏢UpStageYongin-si · South Korea7 works · 728 citations🎓Seoul National University of Science and TechnologySeoul · South Korea2 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Tokenization & segmentation · 124 works

Cite

Cite as Inho Won, Hangyeol Yoo, Minkyung Cho, Jungyeul Park, Hoyun Song and KyungTae Lim. 2026. TReX: Tokenizer Regression for Optimal Data Mixture. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.298

@inproceedings{won2026trex,
  title = {{TReX: Tokenizer Regression for Optimal Data Mixture}},
  author = {Inho Won, Hangyeol Yoo, Minkyung Cho, Jungyeul Park, Hoyun Song and KyungTae Lim},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.298},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.298}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diana Abagyan, Alejandro R Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2025. One tokenizer to rule them all: Emergent language plasticity via multilingual tokenizers. arXiv preprint arXiv:2506.10766. find
  2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. arXiv preprint arXiv:2305.13707.
  3. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Ste… — Tokenizer Choice For LLM Training: Negligible or Crucial?
  4. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  5. Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
  6. Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393. — [arXiv:2310.15393](https://arxiv.org/abs/2310.15393) find
  7. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. arXiv preprint arXiv:2403.06265. find
  8. Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. arXiv preprint arXiv:2004.02334.
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference attack: Bpe tokenizers reveal training data compositions. Advances in Neural Information Processing Systems, 37:8956–8983. find
  11. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  12. N J Karthika, Maharaj Brahma, Rohit Saluja, Ganesh Ramakrishnan, and Maunendra Sankar Desarkar. 2025. Multilingual tokenization through the lens of indian languages: Challenges and insights. find
  13. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30. find
  14. Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junhun Yuk, Changsu Choi, Dongjae Shin, Huije Lee, Hoyun Song, Alice Oh, and Kyungtae Lim. 2025. Kormo: Korean open reasoning model for everyone. find
  15. Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2025. Causal Estimation of Tokenisation Bias. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28325–28340, Vienna, Austria. Association for Comp… find
  16. Yuan Li, Zhengzhong Liu, and Eric P. Xing. 2025. Data mixing optimization for supervised fine-tuning of large language models. In Forty-second International Conference on Machine Learning. find
  17. Jiayu Lin. 2016. On the dirichlet distribution. Department of Mathematics and Statistics, Queens University, 40. find
  18. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025a. SuperBPE: Space travel for language models. In Second Conference on Language Modeling. find
  19. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2025b. Regmix: Data mixture as regression for language model pre-training. In International Conference on Learning Representations (ICLR).
  20. Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. 2025. Enhancing multilingual llm pretraining with model-based data selection. arXiv. find
  21. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. find
  22. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).
  23. Kundeshwar Pundalik, Piyush Sawarkar, Nihar Sahoo, Abhishek Shinde, Prateek Chanda, Vedant Goswami, Ajay Nagpal, Atul Singh, Viraj Thakur, Vijay De-wane, Aamod Thakur, Bhargav Patel, Smita Gautam, Bhagwan Panditi, Shyam Pawar, Madhav Kotcha, Suraj Racha, Saral Sureka, Pankaj Singh, Rishi Bal, Rohit… find
  24. Varshini Reddy, Craig W Schmidt, Yuval Pinter, and Chris Tanner. 2025. How much is enough? the diminishing returns of tokenization training data. arXiv preprint arXiv:2502.20273. — [arXiv:2502.20273](https://arxiv.org/abs/2502.20273) find
  25. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Luccioni, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100. find
  26. Jean Seo, Jaeyoon Kim, SungJoo Byun, and Hyopil Shin. 2025. How does a language-specific tokenizer affect llms? arXiv preprint arXiv:2502.12560. find
  27. Marco Siino, Ilenia Tinnirello, and Marco La Cascia. 2024. Is text preprocessing still worth the time? A comparative survey on the influence of popular preprocessing methods on Transformers and traditional classifiers. Information Systems, 121(102342):1–19. find
  28. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. 2025. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation…
  29. Felix Stollenwerk. 2023. Training and evaluation of a multilingual tokenizer for gpt-sw3. CoRR, abs/2304.14780. — [arXiv:2304.14780](https://arxiv.org/abs/2304.14780) find
  30. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  31. Aamod Thakur, Ajay Nagpal, Atharva Savarkar, Kundeshwar Pundalik, Siddhesh Dosi, Piyush Sawarkar, Viraj Thakur, Rohit Saluja, Maunendra Sankar Desarkar, and Ganesh Ramakrishnan. 2025. The art of breaking words: Rethinking multilingual tokenizer design. arXiv preprint arXiv:2508.06533. find
  32. A Tsun. 2020. Probability and statistics with applications to computing. find
  33. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473–482, Online. Association for Computational Linguistics… find
  34. Philip Whittington, Gregor Bachmann, and Tiago Pimentel. 2025. Tokenisation is NP-Complete. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28133–28153, Vienna, Austria. Association for Computational Linguistics.
  35. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818. find
  36. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. find
  37. Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th biennial conference of the Association for Machine Tran… find
  38. Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, et al. 2025. Falcon-h1: A family of hybrid-head language models redefining efficiency and performance. arXiv preprint arXiv:2507.22448.
  39. Detailed Description of Datasets Subset name Language name rus_Cyrl cmn_Hani deu_Latn spa_Latn jpn_Jpan fra_Latn ita_Latn por_Latn pol_Latn nld_Latn ind_Latn tur_Latn ces_Latn fas_Arab hun_Latn swe_Latn ell_Grek dan_Latn vie_Latn Russian Chinese German Spanish Japanese French Italian Portuguese Poli… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/won-2026-trex-tokenizer-regression-optimal-data/