Elamur Research⌕
☰
home / papers / 2026 / SPQ: An Ensemble Technique for Large Language Model Compression

SPQ: An Ensemble Technique for Large Language Model Compression

Summary

Large language models are costly to store and run, which makes them hard to deploy on memory-limited hardware. Yao and Gultepe propose SPQ, a modular pipeline that gives each part of a transformer its own compression method instead of relying on a single technique.

  • Attention projections are factorized with SVD, keeping enough singular components to retain a chosen share of variance.
  • MLP layers lose whole neurons, ranked by their activation magnitude on calibration data, with a pruning ratio set separately for each layer.
  • All linear layers are then stored as 8-bit symmetric weights without calibration, using per-tensor, per-channel or hybrid scaling.

The authors tested SPQ on OPT, LLaMA, Vicuna and Mistral models. At equal compression ratios, it achieves better perplexity than SVD, pruning or quantization used alone. On LLaMA-2-7B it shrinks memory by up to 75% while perplexity actually improves. It also rivals GPTQ and SparseGPT while using less memory and running faster. The study suggests that simple, layer-aware combinations of standard techniques can make LLMs practical for constrained deployment.

Key points

  • SPQ applies variance-retained SVD to attention layers, activation-based structured pruning (L1/L2 neuron magnitudes; linear, log-inverse or sigmoid ratio normalization) to MLP layers, and post-training 8-bit linear quantization to all linear layers.
  • The hybrid quantization modes are LNH (per-channel for attention, per-tensor for MLP), PBH (percentile-based) and MSH (mean±std threshold on layer sensitivity).
  • On LLaMA-2-7B, memory drops by up to 75%, and WikiText-2 perplexity improves from 5.47 to 4.91. Results on C4, TruthfulQA and GSM8K are preserved.
  • Compared with GPTQ, SPQ needs 6.86 GB versus 7.16 GB and delivers up to 1.9× higher inference throughput.
  • The experiments cover models from 1B to 7B parameters. Code is released at github.com/JiaminYao/SPQ_LLM_Compression.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3t98siww4xf2

@inproceedings{yao2026spq,
  title = {{SPQ: An Ensemble Technique for Large Language Model Compression}},
  author = {Yao, Jiamin and Gultepe, Eren},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3t98siww4xf2},
  url = {https://doi.org/10.63317/3t98siww4xf2}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Banner, Yury Nahshan, and Daniel Soudry. 2019. Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in neural information processing systems, 32. find
  2. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. find our card
  3. Kamran Chitsaz, Quentin Fournier, Goncalo Mordido, and Sarath Chandar. 2024. Exploring quantization for efficient pre-training of transformer language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13473–13487, Miami, Florida, USA. Association for Computation… find
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  6. Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27. find
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115. find
  8. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. find
  9. Yong Guan, Xiaozhi Wang, Lei Hou, Juanzi Li, Jeff Pan, Jiaoyan Chen, and Freddy Lecue. 2024. Tacoere: Cluster-aware compression for event relation extraction. arXiv preprint arXiv:2405.06890. find
  10. Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. 2025. Slimllm: Accurate structured pruning for large language models. arXiv preprint arXiv:2505.22689. — [arXiv:2505.22689](https://arxiv.org/abs/2505.22689) find
  11. Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. find
  12. Yang He and Lingao Xiao. 2023. Structured pruning for deep convolutional neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(5):2900–2919. find
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  14. Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. 2016. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250. — [arXiv:1607.03250](https://arxiv.org/abs/1607.03250) find
  15. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recogni… find
  16. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  17. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710. — [arXiv:1608.08710](https://arxiv.org/abs/1608.08710) find
  18. Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han. 2024. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007. find
  19. Wei Li, Lujun Li, You-Liang Huang, Mark G Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025a. Structured mixture-of-experts llms compression via singular value decomposition. find
  20. Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. 2025b. Adasvd: Adaptive singular value decomposition for large language models. arXiv e-prints, pages arXiv–2502. find
  21. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100. find
  22. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. find our card
  23. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720. find our card
  24. Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, and Egor Anatolievich Filimonov. 2024. On the way to lossless compression of language transformers: Exploring cross-domain properties of quantizat… find
  25. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. find
  26. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. find
  28. Siyu Ren and Kenny Q Zhu. 2023. Low-rank prune-and-factorize for language model compression. arXiv preprint arXiv:2306.14152. find
  29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  30. Moritz Thoma, Jorge Villasante, Emad Aghajanzadeh, Shambhavi Balamuthu Sampath, Pierpaolo Mori, Maximilian Groetzinger, Daniil Dylkin, Manoj-Rohit Vemparala, Nael Fasfous, Alexander Frickenstein, et al. 2025. Flar-svd: Fast and latency-aware singular value decomposition for model compression. In Pro… find
  31. Pingjie Wang, Hongcheng Liu, Yanfeng Wang, and Yu Wang. 2024a. Pruning before fine-tuning: A retraining-free compression framework for pre-trained language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2… find
  32. Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. 2025a. Dobi-svd: Differentiable svd for llm compression and some new perspectives. arXiv preprint arXiv:2502.02723. — [arXiv:2502.02723](https://arxiv.org/abs/2502.02723) find
  33. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. find
  34. Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. 2025b. Svd-llm v2: Optimizing singular value truncation for large language model compression. arXiv preprint arXiv:2503.12340. find
  35. Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024b. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378. — [arXiv:2403.07378](https://arxiv.org/abs/2403.07378) find
  36. Yongchang Wang and Ligu Zhu. 2017. Research and implementation of svd in machine learning. In 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS), pages 471–475. IEEE. find
  37. Hong Yankun, Li Xing, Zhen Hui-Ling, Yu Xianzhi, Liu Wulong, and Yuan Mingxuan. 2025. Svdq: 1.25-bit and 410x key cache compression for llm attention. arXiv preprint arXiv:2502.15304. — [arXiv:2502.15304](https://arxiv.org/abs/2502.15304) find
  38. Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) find
  39. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36–39. IEEE. find
  40. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-spq-ensemble-technique-large-language/