Elamur Research⌕
☰
home / papers / 2026 / LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model

LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model

Paper · 2026 · Wei Shao, Zheng Lingchao, Pengyu Wang et al.

Summary

LoPT targets an often-ignored source of latency in long-context LLM inference: tokenization of very long inputs, which runs on the CPU. Existing chunk-and-parallelize approaches, whether they split on delimiters or use overlapping chunks like ParallelTokenizer, can produce token sequences that differ from ordinary sequential tokenization. This happens because tokens change near chunk boundaries, and token-ID overlap matching fails on repetitive text. LoPT keeps the split, parallel tokenize and merge pipeline but changes how chunks are stitched together. Each chunk's tokens carry their character offsets, and the overlap between neighbouring chunks is located by checking that both chunks map those tokens to the same global character positions. If the detected overlap is too short, the chunk length is increased and the process reruns. The authors give a formal argument that the output is lossless. Experiments on LongBenchV2, LEval (medium subsets) and CLongEval show identical outputs with substantial speedups over HuggingFace TokenizerFast. This matters for latency-sensitive, small-batch inference.

Key points

  • The framework has three modules: text split (fixed chunk length L_c plus overlap L_o), multiprocess parallel tokenization returning token IDs with character positions, and a position-aware match-and-merge step. A dynamic chunk-length mechanism guarantees a sufficiently long overlap.
  • The method is evaluated on LongBenchV2, LEval (medium) and CLongEval, which cover several domains, formats and languages, using tokenizers such as Qwen3 and DeepSeek-V3. Outputs are fully consistent with standard tokenization.
  • On LongBenchV2 with the DeepSeek-V3 tokenizer, speedup grows from 1.70× with 2 processes to 7.27× with 32. Amdahl's Law predicts 29.85× (serial fraction 0.000935), and the authors attribute the gap to inter-process communication and scheduling overhead.
  • LoPT beats a sample-level multiprocessing HF tokenizer at small batch sizes, but the advantage narrows as batch size grows. It is also less sensitive to CPU strength, with larger gains on weaker CPUs.
  • There are two limitations: the base tokenizer must expose character offsets, and the method is not intended for large-batch processing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHardware and Architecture · 81 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksTokenization & segmentation · 119 works

Cite

Cite as Wei Shao, Zheng Lingchao, Pengyu Wang, Peizhen Zheng, Li Jun and Yuwei Fan. 2026. LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1529

@inproceedings{shao2026lopt,
  title = {{LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model}},
  author = {Wei Shao, Zheng Lingchao, Pengyu Wang, Peizhen Zheng, Li Jun and Yuwei Fan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1529},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1529}
}

References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ultra-large capacity on-chip memories. Preprint, arXiv:2508.08457. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang,… — Large Language Model Agent: A Survey on Methodology, Applications and Challenges
  2. OpenAI. 2022. tiktoken: a fast BPE tokeniser for use with OpenAI’s models. GitHub repository. Accessed: 2026-04-18. find
  3. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. Preprint, arXiv:2307.11088.
  4. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  5. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. Preprint, arXiv:2412.15204.
  6. David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2022. The carbon footprint of machine learning training will plateau, then shrink. Preprint, arXiv:2204.05149. — [arXiv:2204.05149](https://arxiv.org/abs/2204.05149) find
  7. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Preprint, arXiv:2205.14135. find
  8. Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. Preprint, arXiv:2403.03514. find
  9. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  10. RAPIDS. 2019. cuDF - GPU DataFrame Library. GitHub repository. Accessed: 2026-04-18. find
  11. DeepSeek-AI. 2025. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention. find
  12. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  14. Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. Preprint, arXiv:2312.00752. find
  15. Hugging Face. 2018. Tokenizers: Fast State-of-the-Art Tokenizers optimized for Research and Production. GitHub repository. Accessed: 2026-04-18. find
  16. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  17. Ming-Yen Lee, Faaiq Waqar, Hanchen Yang, Muhammed Ahosan Ul Karim, Harsono Simka, and Shimeng Yu. 2025. Architecting long-context llm acceleration with packing-prefetch scheduler and OpenLLMab. 2024. ParallelTokenizer: Use the tokenizer in parallel to achieve superior acceleration. GitHub repository… find
  18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  19. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025. Kimi k2: Open agentic intelligence. Preprint, a…
  20. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural machine translation with byte-level subwords. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9154–9160.
  21. Renjun Xu and Jingwen Peng. 2025. A comprehensive survey of deep research: Systems, methodologies, and applications. Preprint, arXiv:2506.12594. — [arXiv:2506.12594](https://arxiv.org/abs/2506.12594) find
  22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  23. Amos You. 2025. Blockbpe: Parallel bpe tokenization. Preprint, arXiv:2507.11941. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shao-2026-lopt-lossless-parallel-tokenization-acceleration/