Elamur Research⌕
☰
home / papers / 2026 / Resonating with RoPE: Spectral Quantization for High-Fidelity Key Cache Compression

Resonating with RoPE: Spectral Quantization for High-Fidelity Key Cache Compression

Paper · 2026 · Xuefei Wang, Haoyu Tang, Tianyuan Liang et al.

Summary

In long-context LLM inference, the KV cache grows linearly with sequence length and can outweigh the model weights in memory. Key vectors are especially hard to quantize because RoPE rotates channel pairs by position. This creates periodic swings along the token axis that widen the value range and coarsen low-bit quantization. The authors observe that applying a DCT along the token dimension turns these swings into a sparse spectrum. Most energy sits in a few neighbouring coefficients, and the peak location is stable across layers and heads.

Their framework, SpectrumQuant, has three parts:

  • it stores the dominant spectral peak at high precision;
  • it assigns different bit-widths to the low- and high-frequency residual bands;
  • it amplifies high frequencies so all bands can share one quantizer.

Fused Triton kernels defer the inverse transform and accumulate sparse terms on-chip, which removes the latency cost of the transform. No offline calibration is needed. On LongBench and other benchmarks, accuracy and latency stay close to FP16 while peak GPU memory drops noticeably.

Key points

  • Theory: RoPE shifts each channel's spectral centre to a predictable frequency, k* = Gθ_c/π. The two DCT coefficients nearest to it are proven to hold at least about 61% (6/π²) of the energy.
  • Ablations: The DCT version clearly beats a time-domain baseline. Keeping N = 2 dominant coefficients recovers most of the accuracy, and the pre-emphasis factor σ works stably between 2.0 and 3.0.
  • Results on Qwen3-8B-AWQ (LongBench): At about 3.71 bits, the average score is 38.68, compared with 38.88 for FP16. This shows the method is compatible with AWQ weight quantization.
  • Robustness: Results hold across different RoPE setups:
    • Llama-2-7B-32K (linear scaling);
    • Llama-3.1-8B-Instruct (b = 500,000, YaRN-like scaling);
    • the Qwen3 series (b = 1,000,000).
  • Value cache: The method combines with KIVI-style token-wise value quantization, down to 2 bits, with acceptable accuracy loss.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Shandong UniversityJinan · China17 works · 789 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksHardware and Architecture · 81 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Xuefei Wang, Haoyu Tang, Tianyuan Liang, Zhibin Wang, Yupeng Hu and Weili Guan. 2026. Resonating with RoPE: Spectral Quantization for High-Fidelity Key Cache Compression. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1732

@inproceedings{wang2026resonating,
  title = {{Resonating with RoPE: Spectral Quantization for High-Fidelity Key Cache Compression}},
  author = {Xuefei Wang, Haoyu Tang, Tianyuan Liang, Zhibin Wang, Yupeng Hu and Weili Guan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1732},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1732}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895– 4…
  3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
  4. Ningning Chen, Weicai Ye, and Ying Jiang. 2025. Hbllm: Wavelet-enhanced high-fidelity 1-bit quantization for llms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  5. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. find
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  7. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  8. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  9. Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. 2024b. Parameter-efficient fine-tuning with discrete fourier transform. In International Conference on Machine Learning, pages 14884–14901. PMLR. gkamradt. 2023. Llmtest needle in a haystack - pressure testing llms.… find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  11. Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Preprint, arXiv:2405.14256. find
  12. Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, and Zhouhan Lin. 2023. Fourier transformer: Fast long range modeling by removing sequence redundancy with FFT operator. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8954–8966, Toronto, Canad… find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  14. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Preprint, arXiv:2401.18079.
  15. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  16. Jushi Kai, Boyi Zeng, Yixuan Wang, Haoli Bai, Ziwei He, Bo Jiang, and Zhouhan Lin. 2025. Freqkv: Frequency domain key-value compression for efficient context window extension. Preprint, arXiv:2505.00570. — [arXiv:2505.00570](https://arxiv.org/abs/2505.00570) find
  17. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  18. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. 2022. FNet: Mixing tokens with Fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4296–4313, Seattle, United…
  19. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100. find
  20. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024a. Spinquant: Llm quantization with learned rotations. In The Thirteenth International Conference on Learning Representations. find
  21. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024b. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, pages 32332–32344.
  22. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
  23. Iván López-Espejo, Aditya Joglekar, Antonio M. Peinado, and Jesper Jensen. 2024. On speech pre-emphasis as a simple and inexpensive method to boost speech enhancement. Preprint, arXiv:2401.09315. — [arXiv:2401.09315](https://arxiv.org/abs/2401.09315) find
  24. Meta. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. find
  25. Alan V Oppenheim, Alan S Willsky, and Syed Hamid Nawab. 1997. Signals & systems. Pearson Educación. find
  26. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. find
  27. Jorn Peters, Marios Fournarakis, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. 2023. Qbitopt: Fast and accurate bitwidth reallocation during training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1282–1291. find
  28. Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. 2025. Beyond position: the emergence of wavelet-like properties in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6074–6088. find
  29. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864.
  30. Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations.
  31. Songhao Wu, Ang Lv, Xun Zhang, Guojun Yin, Wei Lin, and Rui Yan. 2025. Polarquant: Leveraging polar transformation for key cache quantization and decoding acceleration. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  33. Zhixiong Zhao, Fangxin Liu, Junjie Wang, Chenyang Guan, Zongwu Wang, Li Jiang, and Haibing Guan. 2026. Specquant: Spectral decomposition and adaptive truncation for ultra-low-bit llms quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 28786–28794. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-resonating-rope-spectral-quantization-high/