Elamur Research⌕
☰
home / papers / 2026 / SCVQ: Sparse-Compensated Vector Quantization for Large Language Models

SCVQ: Sparse-Compensated Vector Quantization for Large Language Models

Paper · 2026 · Zixuan Zhou, Yujun Diao, Zicheng Kong et al.

Summary

Vector quantization (VQ) compresses LLM weights well at very low bit-widths, but it is slow in practice. Large codebooks and frequent table lookups consume memory bandwidth, and implicit-codebook alternatives depend on costly transforms such as Hadamard rotations. The authors propose SCVQ, a post-training, weight-only method built on standard K-means VQ. It works in four steps:

  • Weights are bi-normalized across rows and columns.
  • Clustering uses K-means weighted by a combined local (activation-based) and global (gradient-based) salience score.
  • Symmetry constraints shrink the codebook.
  • Outliers, highly salient sub-vectors and the largest quantization residuals are stored together in a single sparse correction matrix.

The authors observe that such important weights tend to sit contiguously within rows, which matches VQ's block grouping. They exploit this with a structured sparse format (VSR) and custom CUDA kernels. Normalization scales and sparse terms are then fine-tuned. On the LLaMA-2 family at about 2 bits per weight, SCVQ outperforms AQLM, QuIP# and VPTQ on perplexity and most zero-shot tasks, and it also runs faster. This shows that algorithm–hardware co-design can make aggressive LLM compression practical.

Key points

  • LLaMA-2-7B at 2-bit (2.04 BPW):
    • WikiText-2 perplexity is 5.87, versus roughly 6.13–6.19 for AQLM, QuIP# and VPTQ (FP16: 5.12).
    • C4 perplexity is 7.71, versus about 8.07–8.16 for the baselines.
  • On an NVIDIA A100, SCVQ gives a 1.4× end-to-end speedup over the baseline. LLaMA-2-7B at 2-bit reaches 235.4 tokens/s, 2.3× faster than FP16.
  • Codebook compression cuts the codebook to a quarter of its size, raising perplexity by only 0.13.
  • In ablations, fine-tuning, salience-weighted K-means and the outlier sparse matrix contribute most to quality.
  • Limitations:
    • Experiments use only sub-vector dimension d = 8 and K-means clustering.
    • Sub-vector selection relies on heuristic ranking.
    • The hyperparameter α was tuned by linear search.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing University of Posts and TelecommunicationsBeijing · China78 works · 33,365 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Zixuan Zhou, Yujun Diao, Zicheng Kong, Dehua Ma, Zhenbo Xu, Pei Pei Li and Zhaofeng He. 2026. SCVQ: Sparse-Compensated Vector Quantization for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.403

@inproceedings{zhou2026scvq,
  title = {{SCVQ: Sparse-Compensated Vector Quantization for Large Language Models}},
  author = {Zixuan Zhou, Yujun Diao, Zicheng Kong, Dehua Ma, Zhenbo Xu, Pei Pei Li and Zhaofeng He},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.403},
  url = {https://doi.org/10.18653/v1/2026.acl-long.403}
}

References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Arthur and Sergei Vassilvitskii. 2007. kmeans++: The advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1027–1035. Society for Industrial and Applied Mathematics. of large language models with guarantees. Advances in N… find
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  3. PeterE. Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: Artificial Intelligence,arXiv: Artificial Intelligence.
  4. Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training data. https://github.com/togethercomputer/ RedPajama-Data. Accessed: 2026-01-02. ↗
  5. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318– 30332. find
  6. Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. Preprint, arXiv:2306.03078. find
  7. Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. 2024. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631. find
  8. Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118.
  9. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. find
  10. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer, New York. find
  11. Georgi Gerganov and contributors. 2023. llama.cpp: Inference of llama model in c/c++. https://github.com/ggerganov/llama.cpp. Accessed: 2025-12-27. ↗
  12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  13. Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2022. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291–326. Chapman and Hall/CRC. find
  14. Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. 2023. Quip: 2-bit quantization RobertM. find
  15. Gray. 1984. Vector quantization. IEEE Assp Magazine,IEEE Assp Magazine. find
  16. Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/ accelerate. ↗
  17. Wengang Guo, Kaiyan Lin, and Wei Ye. 2021. Deep embedded k-means clustering. In 2021 International Conference on Data Mining Workshops (ICDMW), pages 686–694. IEEE. find
  18. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  19. Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. Squeezellm: Dense-and-sparse quantization. In International Conference on Machine Learning, pages 23901–23923. PMLR.
  20. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100. find
  21. NVIDIA Corporation. 2024. cuSPARSE Library. https://developer.nvidia.com/cusparse. Accessed: 2024-05-20. ↗
  22. A. Pinar and M.T. Heath. 1999. Improving performance of sparse matrix-vector multiplication. In SC ’99: Proceedings of the 1999 ACM/IEEE Conference on Supercomputing, pages 30–30. find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  24. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, page 8732–8740. find
  25. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137.
  26. S. Tata and J.M. Patel. 2003. Piqa: an algebra for querying protein data sets. In 15th International Conference on Scientific and Statistical Database Management, 2003. find
  27. Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024a. Vptq: Extreme low-bit vector post-training quantization for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8181–8196. find
  28. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  29. Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2024b. Llm-qat: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pag…
  30. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip \#: Even better llm quantization with hadamard incoherence and lattice codebooks. In International Conference on Machine Learning, pages 48630–48656. PMLR.
  31. J. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. find
  32. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv: Computation and Language,arXiv: Computation and Language.
  33. Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. 2025. Gptvq: The blessing of dimensionality for llm quantization. Preprint, arXiv:2402.15319. find
  34. Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In International conference on machine learning, pages 7197–7206. PMLR. find
  35. Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295. — [arXiv:2106.08295](https://arxiv.org/abs/2106.08295) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-scvq-sparse-compensated-vector-quantization/