Elamur Research⌕
☰
home / papers / 2026 / EQUIP: EQUivariant preserving In-Place updates for Efficient Token Pruning

EQUIP: EQUivariant preserving In-Place updates for Efficient Token Pruning

Paper · 2026 · Arun Ramachandran, R Govindarajan, Murali Annavaram et al.

Summary

KV-cache token pruning methods such as StreamingLLM and Heavy Hitters (H2O) keep the cache small, but each eviction leaves gaps. Closing those gaps means shifting and copying whole K/V tensors. Evictions also change token positions, which forces RoPE to be recomputed. EQUIP, from AMD, IISc and USC, removes both costs. It relies on the fact that the attention pipeline (batched matmul, softmax, batched matmul) gives the same result regardless of where a token sits in the cache. Because of this, a new token can be written directly into the slot of the evicted one, with no shifting. For RoPE, EQUIP stores the rotated-key tensors, updates them in the same in-place way, and uses a cheap position reindexing scheme.

The authors tested EQUIP in Hugging Face and vLLM (including Paged Attention) on Intel CPUs and AMD Instinct MI210 GPUs. It gives clear end-to-end speedups and produces exactly the same outputs as the pruning method it accelerates. It also makes token pruning practical inside custom kernels like Paged Attention, which normally struggle with a non-contiguous cache.

Key points

  • Speedups over existing pruning methods (geometric mean):
    • Over StreamingLLM: 1.62× on CPU and 1.47× on GPU.
    • Over H2O: 3.45× on CPU and 1.86× on GPU.
  • Speedups with Paged Attention: combined with vLLM's Paged Attention, EQUIP is 4.18× faster on CPU and 2.61× faster on GPU than autoregressive baselines.
  • Accuracy is unchanged:
    • Perplexity on Llama 2-7B matches StreamingLLM exactly.
    • ARC-Easy accuracy in a streaming setup is 74.90% for both.
    • Layer-level deviations stay below 10⁻⁹ at SDPA and below 10⁻⁵ at RoPE.
  • Multi-token eviction and speculative decoding: EQUIP handles both. With speculative decoding (γ=8) it is 1.47×–1.67× faster than StreamingLLM.
  • Gains depend on batch size and sequence length:
    • At batch size 1, gains are small (about 1.04–1.07×), because MLP computation dominates the runtime.
    • Gains grow with batch size and sequence length, reaching 2.45× at BS=8 with 4k-token sequences on Llama 2-7B.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓Indian Institute of Science BangaloreBengaluru · India10 works · 1,142 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Arun Ramachandran, R Govindarajan, Murali Annavaram and Prakash Raghavendra. 2026. EQUIP: EQUivariant preserving In-Place updates for Efficient Token Pruning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1210

@inproceedings{ramachandran2026equip,
  title = {{EQUIP: EQUivariant preserving In-Place updates for Efficient Token Pruning}},
  author = {Arun Ramachandran, R Govindarajan, Murali Annavaram and Prakash Raghavendra},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1210},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1210}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. 2025. Identify critical kv cache in llm inference from an output perturbation perspective. Preprint, arXiv:2502.03805. Submitted on 6 Feb 2025. — [arXiv:2502.03805](https://arxiv.org/abs/2502.03805) find
  2. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations (ICLR).
  3. Gemini Team and Google. 2023. Gemini: A family of highly capable multimodal models.
  4. Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, Éric de la Clergerie, and Benoît Sagot. 2025. Q-filters: Leveraging qk geometry for efficient kv cache compression. Preprint, arXiv:2503.02812. Submitted on 4 Mar 2025. — [arXiv:2503.02812](https://arxiv.org/abs/2503.02812) find
  5. HuggingFace. 2024. Transformers: State-of-the-art natural language processing for pytorch, tensorflow, and jax. Accessed: 2024-07-06. find
  6. A. H. Hunter, Chris Kennelly, Paul Turner, Darryl Gove, Tipp Moseley, and Parthasarathy Ranganathan. 2021. Beyond malloc efficiency to fleet efficiency: a hugepage-aware memory allocator. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 257–273. find
  7. Intel. 2024a. The AI PC opportunity. Intel White Paper. Originally available at https://www.intel.com/content/dam/www/central-libraries/ us/en/documents/2024-01/ the-ai-pc-opportunity-white-paper.pdf; accessed January 2024. ↗
  8. Intel. 2024b. Mkl: Improved small matrix performance using just-in-time (jit) code. Accessed: 2024-08-01. find
  9. Intel Corporation. 2024. oneDNN API Documentation. Accessed: 2024-10-31. find
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  11. Risi Kondor and Shubhendu Trivedi. 2018. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2747–2755. find
  12. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP). ACM.
  13. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), pages 19274–19286. find
  14. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024a. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS).
  15. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE-2: Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing (EMNLP).
  16. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024c. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning (ICML).
  17. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. find
  18. Manlai Liang, JiaMing Zhang, Xiong Li, and Jinlong Li. 2025. LagKV: Lag-relative information of the KV cache tells which tokens are important. Preprint, arXiv:2504.04704. — [arXiv:2504.04704](https://arxiv.org/abs/2504.04704) find
  19. Linux Kernel Community. 2024. Transparent hugepages – Linux kernel documentation. https://docs.kernel.org/admin-guide/mm/transhuge.html. Accessed: 2024-07-21. ↗
  20. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning (ICML). llama. 2023. Llama-7b: A large language model, hugging face and hug…
  21. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems.
  22. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. find
  23. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). OpenBookQA dataset: https://huggingface.co… ↗
  24. MLCommons. 2024. Inference data center benchmarks. Accessed: 2024-08-01. find
  25. NVIDIA. 2022. Fastertransformer. https://Ac-github.com/NVIDIA/FasterTransformer. cessed: 2024-08-01. ↗
  26. NVIDIA. 2023. Tensorrt-llm. Accessed: 2024-08-02. find
  27. NVIDIA. 2025a. Chunkkvpress. https://github.com/NVIDIA/kvpress/blob/main/ kvpress/presses/chunkkv_press.py. ↗
  28. NVIDIA. 2025b. Chunkpress. https://github.com/NVIDIA/kvpress/blob/main/ kvpress/presses/chunk_press.py. ↗
  29. NVIDIA. 2025c. Expectedattention. https://github.com/NVIDIA/kvpress/blob/main/kvpress/ presses/expected_attention_press.py. ↗
  30. NVIDIA. 2025d. Finchpress. https://github.com/ NVIDIA/kvpress/blob/main/kvpress/presses/ finch_press.py. ↗
  31. NVIDIA. 2025e. Keyrerotationpress. https://github.com/NVIDIA/kvpress/blob/main/ kvpress/presses/key_rerotation_press.py. ↗
  32. NVIDIA. 2025f. Observedattention. https://github.com/NVIDIA/kvpress/blob/main/kvpress/ presses/observed_attention_press.py. ↗
  33. NVIDIA. 2025g. Randompress. https://github.com/NVIDIA/kvpress/blob/main/ kvpress/presses/random_press.py. ↗
  34. NVIDIA. 2025h. Scorerpress. https://github.com/NVIDIA/kvpress/blob/main/ kvpress/presses/scorer_press.py. ↗
  35. NVIDIA. 2025i. Tova. https://github.com/ NVIDIA/kvpress/blob/main/kvpress/presses/ tova_press.py. ↗
  36. S. Park, J. Choi, S. Lee, and U. Kang. 2024. A comprehensive survey of compression algorithms for language models. arXiv preprint. find
  37. Aaron Pham, Chaoyu Yang, Sean Sheng, Shenyang Zhao, Sauyon Lee, Bo Jiang, Fog Dong, Xipeng Guan, and Frost Ming. 2023. Openllm: Operating llms in production. find
  38. Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic memory management for serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (AS… find
  39. PyTorch. 2024. Pytorch serve. Accessed: 2024-08-01. find
  40. Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2025. MagicDec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In International Conference on Learning Repr… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramachandran-2026-equip-equivariant-preserving-place-updates/