Elamur Research⌕
☰
home / papers / 2026 / SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Models

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Models

Summary

Frequency-driven tokenizers such as BPE and WordPiece split text without regard to meaning. In long inputs, they spend many tokens on repetitive or boilerplate material, which enlarges the KV cache and slows attention. SemToken addresses this by compressing the token stream before it reaches the model. A frozen lightweight encoder (e.g., SimCSE or distilled BERT) embeds tokens in sliding windows. Adjacent tokens with high cosine similarity are greedily merged. A covariance-based "semantic entropy" score then decides whether each span is kept fine-grained or coarsened. Under a fixed budget, the highest-entropy spans are retained. During generation, a query-conditioned step drops past spans with low relevance to the current query. The method needs no retraining and works with any model. The authors test it on WikiText-103, BookSum, LongBench and ChartQA across several LLMs. They report roughly halved latency and a much smaller memory footprint with perplexity close to the baseline. The work matters because it treats tokenization itself as a lever for long-context efficiency, and its gains stack with attention and cache optimizations.

Key points

  • On WikiText-103, SemToken keeps 41% of BPE's tokens. Latency falls from 61.2 ms to 30.4 ms and memory from 4.1 GB to 1.5 GB. Perplexity is 17.0 versus 17.3 for BPE. It also outperforms the Entropy-Pruned, VQ-Tok and TofuTok baselines.
  • The headline claims are 2.4× token reduction and 1.9× end-to-end speedup. Reported memory savings vary across sections: 67% in the abstract, 62% KV cache reduction in the conclusion, and 58% in the cross-model text.
  • Gains hold across LLaMA-2 7B/70B, GPT-J 6B, Qwen-14B and Mistral-7B. Compression ranges from 2.2× to 2.6×, and perplexity rises by under 1%.
  • Paired with FlashAttention2, the total speedup reaches 2.7×, about 84% of the theoretical ideal. Adding H2O pruning cuts total cache memory by about 79%.
  • In ablations, removing any single module hurts results. The three modules are clustering, density scoring and query-aware AR-budgeting. Without clustering, for example, PPL rises to 17.8 and latency to 37.9 ms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.1

@inproceedings{liu2026semtoken,
  title = {{SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Models}},
  author = {Liu, Dong and Yu, Yanxuan},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.1},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.1}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. find our card
  2. Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, and Xuhong Wang. 2025a. PiKV: KV cache management system for moe architecture. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. find
  3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. find
  4. Dong Liu, Yanxuan Yu, Xuhong Wang, Ben Lengerich, and Ying Nian Wu. 2025b. MKA: Memory-keyed attention for efficient long-context reasoning. In ICML 2025 Workshop on Long-Context Foundation Models. find
  5. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022. Rethinking attention with performers. find
  6. Dong Liu, Yanxuan Yu, Yite Wang, Jing Wu, Zhongwei Wan, Sina Alinejad, Benjamin Lengerich, and Ying Nian Wu. 2025c. Designing large foundation models for efficient training and inference: A survey. find
  7. NVIDIA Corporation. 2024. Cutlass: Cuda templates for linear algebra subroutines. https://github.com/NVIDIA/cutlass. ↗
  8. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. find
  9. Dong Liu, Jiayi Zhang, Yifan Li, Yanxuan Yu, Ben Lengerich, and Ying Nian Wu. 2025d. Fastcache: Fast caching for diffusion transformer through learnable linear approximation. find
  10. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2021. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681. — [arXiv:2111.12681](https://arxiv.org/abs/2111.12681) find
  11. Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with supervised attention. arXiv preprint arXiv:1609.04186. find
  12. Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transformers. find
  13. Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024a. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–56. find
  14. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. Booksum: A collection of datasets for long-form narrative summarization. find
  15. Zeyuan Liu, Ziyu Huan, Xiyao Wang, Jiafei Lyu, Jian Tao, Xiu Li, Furong Huang, and Huazhe Xu. 2024b. World models with hints of large language models for goal achieving. find
  16. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. find
  17. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. find our card
  18. Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. Tr-bert: Dynamic token reduction for accelerating bert inference. arXiv preprint arXiv:2105.11618. find
  19. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. find
  20. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models. find
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/liu-2026-semtoken-semantic-aware-tokenization-efficient/