Elamur Research⌕
☰
home / papers / 2026 / OjaKV: Context-Aware Online Low-Rank KV Cache Compression

OjaKV: Context-Aware Online Low-Rank KV Cache Compression

Paper · 2026 · Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri et al.

Summary

Long-context LLM inference is limited by GPU memory for the key-value (KV) cache. For Llama-3.1-8B with a 32K-token prompt at batch size 4, the cache reaches about 16 GB, roughly the size of the model weights. Earlier low-rank compression methods project keys and values onto a fixed basis learned offline from calibration data. That basis loses accuracy when inference prompts differ from the calibration distribution.

OjaKV takes a two-part approach. First, a hybrid storage policy keeps tokens with high reconstruction error in full rank. Second, the remaining tokens are compressed into a subspace that is updated online with Oja's incremental PCA rule. The update is thorough during prefill and lighter and periodic during decoding. The method needs no fine-tuning, works with FlashAttention and can be combined with token-eviction methods for further savings.

The authors test it on lm-eval-harness, LongBench, RULER and AIME. OjaKV beats static low-rank baselines at the same compression ratios, with the largest gains on long, reasoning-heavy generation tasks.

Key points

  • Compared against static low-rank methods (EigenAttention/Eigen-N, MatryoshkaKV, StaticPCA, Palu). The authors describe this as the first evaluation of projection-based KV compression on generation-centric long-context benchmarks (LongBench, RULER).
  • On AIME2025, static methods (Eigen-N, StaticPCA, Palu) score 0%. OjaKV reaches 13.0%, against 43.3% for the uncompressed model.
  • On Llama-3.1-8B-Instruct at 60% compression and 32K tokens:
    • Memory falls from 16 GB to 11.6 GB.
    • TTFT rises from 2102 ms to 2801 ms.
    • Decoding is 10.9–13.4% slower, depending on the update interval T (64–256 tokens).
  • RULER ablation (Llama-2-7B, 0.6x compression): average accuracy rises from 38.9% (StaticPCA) to 57% with hybrid storage and to 60% with online Oja updates added.
  • Code is available at github.com/zzbright1998/OjaKV. Stated limitations are hyperparameter tuning (learning rates, buffer size, top-k) and extra compute compared with static compression.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Rensselaer Polytechnic InstituteTroy · United States21 works · 4,933 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati and Pin-Yu Chen. 2026. OjaKV: Context-Aware Online Low-Rank KV Cache Compression. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.494

@inproceedings{zhu2026ojakv,
  title = {{OjaKV: Context-Aware Online Low-Rank KV Cache Compression}},
  author = {Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati and Pin-Yu Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.494},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.494}
}

References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anisha Agarwal, Aaron Chan, Shubham Chandel, Jinu Jang, Shaun Miller, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Neel Sundaresan, and Michele Tufano. 2024. Copilot evaluation harness: Evaluating llm-guided software programming. Preprint, arXiv:2402.14261. — [arXiv:2402.14261](https://arxiv.org/abs/2402.14261) find
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
  3. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. Matharena: Evaluating llms on uncontaminated math competitions. find
  4. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, and 1 others. 2024. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782. find
  5. Matthew Brand. 2002. Incremental singular value decomposition of uncertain data with missing values. In European Conference on Computer Vision, pages 707–720. Springer. find
  6. Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. 2024. Palu: Compressing kv-cache with low-rank projection. Preprint, arXiv:2407.21118. find
  7. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Preprint, arXiv:2205.14135. find
  8. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  10. Ming Gu and Stanley C Eisenstat. 1994. A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem. SIAM journal on Matrix Analysis and Applications, 15(4):1266–1276. find
  11. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440–2452. find
  12. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303.
  13. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.
  14. Andrew V Knyazev and Peizhen Zhu. 2012. Principal angles between subspaces and their tangents. arXiv preprint arXiv:1209.0523. — [arXiv:1209.0523](https://arxiv.org/abs/1209.0523) find
  15. Gleb Kumichev, Pavel Blinov, Yulia Kuzkina, Vasily Goncharov, Galina Zubkova, Nikolai Zenovkin, Aleksei Goncharov, and Andrey Savchenko. 2024. Medsyn: Llm-based synthetic medical text generation framework. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 2… find
  16. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970.
  17. Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. 2024. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint arXiv:2410.14731. — [arXiv:2410.14731](https://arxiv.org/abs/2410.14731) find
  18. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, and 1 others. 2024a. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434. find
  19. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024b. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  20. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024c. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750.
  21. Mohammad Najafzadeh and Mohammad Mahmoudi-Rad. 2024. Residual energy evaluation in vortex structures: On the application of machine learning models. Results in Engineering, 23:102792. find
  22. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. find
  23. Yuxuan Zhu, Ali Falahati, David H Yang, and Mohammad Mohammadi Amiri. 2025. Sentencekv: Efficient llm inference via sentence-level semantic kv caching. arXiv preprint arXiv:2504.00970. find
  24. Erkki Oja. 1982. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3):267–273. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-ojakv-context-aware-online-low/