Elamur Research⌕
☰
home / papers / 2026 / Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices

Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices

Paper · 2026 · Liu Zai and Iraklis A. Klampanos

Summary

Byte-level BPE tokenizers start with a pretokenization step that splits text into fragments before pair merging. This step usually relies on a complex regular expression, and it stays sequential. On edge hardware such as PCs, laptops and embedded devices, the regex engine can cost throughput and cold-start time. The authors analyse the cl100k-style pattern used by GPT-3, LLaMa-3 and Qwen-2.5. They show that the character set can be reduced to seven categories. With those categories, the branch a segment belongs to depends only on its first two characters.

Peek2 uses this to replace the regex's try-and-fall-back matching with a single lookup in a 7×7 table. It then calls a simple branch-specific scanning function. The method runs in linear time with constant memory, and its output matches the original pretokenizer exactly, including its quirks. Built in Safe Rust inside Hugging Face tokenizers, Peek2 more than doubles pretokenization speed and gives modest end-to-end encoding gains. No model retraining is needed.

Key points

  • PeekCategorize sorts each Unicode scalar into one of 7 classes, such as ASCII space, quote, CR/LF, and the Letter/Whitespace/Number classes. The table that picks among Branch0–Branch6 is inspired by Hopcroft's DFA minimization.
  • The algorithm runs in O(n) time with O(1) space. NFA regex engines, by comparison, can reach O(n×k) or O(n×m).
  • It is implemented in Safe Rust on Hugging Face tokenizers 0.22.3-dev.0 and tested on an Intel Core i5-13600KF. The four test datasets are FineWeb (en), Fineweb-Edu-Chinese (cn), codesearchnet (code) and opc-fineweb-math (math).
  • Pretokenization alone speeds up by 2.20× (cn) to 2.48× (en). Full LLaMa-3 encoding improves by 1.03–1.14×, while gains in BPE training are smaller.
  • Pretokenization's share of encoding time drops sharply, for example from 22.8% to 14.0% on code. Experiments are limited to desktop CPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GlasgowGlasgow · United Kingdom14 works · 746 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksTokenization & segmentation · 124 works

Cite

Cite as Liu Zai and Iraklis A. Klampanos. 2026. Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.10

@inproceedings{zai2026peek2,
  title = {{Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices}},
  author = {Liu Zai and Iraklis A. Klampanos},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.10},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.10}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Philip Bille and Mikkel Thorup. 2010. Regular expression matching with multi-strings and intervals. In Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’10, page 1297–1308, USA. Society for Industrial and Applied Mathematics. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  4. Russ Cox. 2007. Regular expression matching can be simple and fast (but is slow in java, perl, php, python, ruby, ...). find
  5. Zixu Hao, Huiqiang Jiang, Shiqi Jiang, Ju Ren, and Ting Cao. 2024. Hybrid slm and llm for edge-cloud collaborative inference. In Proceedings of the Workshop on Edge and Mobile Foundation Models, EdgeFM ’24, page 36–41, New York, NY, USA. Association for Computing Machinery. find
  6. John Hopcroft. 1971. An n log n algorithm for minimizing states in a finite automaton. In Zvi Kohavi and Azaria Paz, editors, Theory of Machines and Computations, pages 189–196. Academic Press. find
  7. Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. 2024. Opencoder: The open cookbook for top-tier code large lang…
  8. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. ArXiv.
  9. Anthony Moi and Nicolas Patry. 2025. HuggingFace’s Tokenizers. find
  10. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
  11. Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Compu…
  12. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  13. Jianshu She, Wenhao Zheng, Zhengzhong Liu, Hongyi Wang, Eric P. Xing, Huaxiu Yao, and Qirong Ho. 2025. Token level routing inference system for edge devices. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 159–166, V… find
  14. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  15. Ken Thompson. 1968. Programming techniques: Regular expression search algorithm. Commun. ACM, 11(6):419–422. find
  16. Amos You. 2025. Blockbpe: Parallel bpe tokenization. Preprint, arXiv:2507.11941. find
  17. Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. 2025. Opencsg chinese corpus: A series of high-quality chinese datasets for llm training. Preprint, arXiv:2501.08197. — [arXiv:2501.08197](https://arxiv.org/abs/2501.08197) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zai-2026-peek2-regex-free-byte-level/