Elamur Research⌕
☰
home / papers / 2026 / SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

Paper · 2026 · Andrei-Valentin Tănase and Elena Pelican

Summary

Most subword tokenizers, including standard BPE, split text on whitespace before learning a vocabulary, so a token can never span more than one word. This lengthens sequences and spends vocabulary slots on fragments. It hurts most with recurring multiword expressions and with scripts that do not mark word boundaries. The authors introduce SupraTok, a modular tokenizer that learns units across those boundaries. It has three components:

  • Data filtering (optional): removes low-information documents based on character-bigram entropy.
  • Three-pass curriculum: ordinary merges come first, then PMI-seeded multiword candidates, then longer expression spans that are kept only if they show net benefit on held-out data.
  • Script-aware handling: Unicode-based processing, plus lightweight heuristics for languages such as Turkish and Finnish.

SupraTok is compared with BPE, SuperBPE and SentencePiece cross-boundary BPE on identical unfiltered public data, and the effect of filtering is measured separately. SupraTok compresses better at every tested vocabulary size, trains faster than SuperBPE, holds up across 14 languages, and helps downstream GPT-2-style models. Better compression lets a model see more text within the same context window, with no architectural changes.

Key points

  • With a 100k vocabulary and matched unfiltered data, SupraTok's compression (characters per token, C/T) is 17.5% higher than standard BPE and 1.8% higher than the official SuperBPE. It also trains 2.1× faster than SuperBPE. Across vocabularies from 50k to 300k, its lead over SuperBPE ranges from 1.8% to 8.6%.
  • The entropy filter keeps 10%, 50% and 90% of low-, medium- and high-entropy documents (thresholds 3.0 and 4.5), reducing a 100k-document English set to 76k documents. Filtering lifts SupraTok from 5.78 to 5.99 C/T and gives SuperBPE a smaller gain. SP-BPE-CrossBoundary is essentially unchanged.
  • On FLORES-200 across 14 languages, SupraTok's macro-averaged relative gain over BPE is 34.9%. Script-aware processing alone adds 11.9% on average across seven languages.
  • Downstream tests used 12L-768d and 24L-1024d GPT-2-style backbones with 256k vocabularies, matched compute and fixed token budgets. SupraTok improved HellaSwag and MMLU without reducing output diversity.
  • In the ablation, the curriculum and cross-boundary learning account for most of the benefit. Seed mining uses n-grams of length 2–6, a PMI threshold of 2.0 and a minimum frequency of 100.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksTokenization & segmentation · 119 works

Cite

Cite as Andrei-Valentin Tănase and Elena Pelican. 2026. SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.774

@article{tanase2026supratok,
  title = {{SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance}},
  author = {Andrei-Valentin Tănase and Elena Pelican},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.774},
  url = {https://doi.org/10.1162/tacl.a.774}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. CANINE: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91.
  2. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
  3. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027v1. find
  4. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409v1.
  5. Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Zhou Xun. 2025. Over-tokenized transformer: Vocabulary is generally worth scaling. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 17913–179… find
  6. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75.
  7. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
  8. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
  9. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025. SuperBPE: Space travel for language models. In Second Conference on Language Modeling. find
  10. John Sinclair. 1991. Corpus, Concordance, Collocation. Oxford University Press. find
  11. Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srini Iyer. 2025. Byte latent transformer: Patches scale better than tokens. In Proceedings of the 6… find
  12. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892.
  13. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  14. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 977–983. find
  15. Craig W. Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter. 2025. Boundless byte pair encoding: Breaking the pre-tokenization barrier. In Second Conference on Language Modeling. find
  16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-cc: Transforming common crawl into a refined long-horizon pr… find
  17. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. Scaling laws with vocabulary: Larger models deserve larger vocabularies. In Advances in Neural Information Processing Systems, volume 37.
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
  19. Sami Virpioja, Peter Smit, Stig-Arne Grönroos, and Mikko Kurimo. 2013. Morfessor 2.0: Python implementation and extensions for Morfessor baseline. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 21–24. find
  20. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473–482. find
  21. Alison Wray. 2002. Formulaic Language and the Lexicon. Cambridge University Press. find
  22. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  23. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. find
  24. Phase 3 Scoring and Acceptance Rules With σ(k, x) = 1/(1 + e −kx ), define find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tanase-2026-supratok-cross-boundary-tokenization-enhanced/