Elamur Research⌕
☰
home / papers / 2026 / Low-Rank Compression of Language Models via Differentiable Rank Selection

Low-Rank Compression of Language Models via Differentiable Rank Selection

Summary

Low-rank (SVD-based) compression of LLMs depends on choosing how many singular values to keep in each layer. Existing approaches have clear weaknesses. STRS is a heuristic that searches a small discrete set of ratios and treats each layer on its own. ARS is gradient-based, but it degrades quality badly unless the model is fine-tuned after compression.

The authors propose Learning to Low-Rank Compress (LLRC). Each targeted weight matrix is factorised with SVD, and a learnable mask, relaxed with a Gumbel sigmoid, is placed over its singular values. Only these mask parameters are trained, using a small Wikitext calibration set. The loss balances two goals: reaching the target compression and keeping intermediate activations close to those of the uncompressed model, which acts as a distillation signal. The learned masks then set per-layer ranks directly, with no post-compression fine-tuning.

On Llama-2, Llama-3 and Gemma models, LLRC usually outperforms other fine-tuning-free rank selectors and compression baselines on common-sense reasoning and open-domain QA. It also comes close to LLM-Pruner with fine-tuning, making cheap compression of LLMs more practical.

Key points

  • Models evaluated: Llama-2-7B, Llama-2-13B, Llama-3-8B and Gemma-7B, at parameter ratios of 0.90, 0.85 and 0.80.
  • Benchmarks: NQ-Open, MMLU, BoolQ, PIQA and OpenbookQA.
  • Baselines: Fixed Rate, STRS, ARS, SVD-LLM and LLM-Pruner.
  • Llama-2-13B at 20% compression: LLRC beats STRS by about 12 points on MMLU (44.1 vs 32.1), 3.5 on BoolQ and 4.4 on OpenbookQA.
  • Gemma-7B at ratio 0.80: LLRC keeps 33.6 MMLU and 75.8 BoolQ, compared with STRS at 23.4 and 51.2.
  • Where STRS wins: STRS still scores higher in some cases, for example Llama-3-8B MMLU at ratio 0.90 (49.7 vs 44.8).
  • Without fine-tuning: LLRC outperforms fine-tuning-free LLM-Pruner across ratios. It is roughly on par with LLM-Pruner fine-tuned on Alpaca.
  • Ablation on the training objective: the activation-distillation objective generalises better than a next-token pre-training objective. At ratio 0.80 it wins on every dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2xbs948bhby9

@inproceedings{sundrani2026low,
  title = {{Low-Rank Compression of Language Models via Differentiable Rank Selection}},
  author = {Sundrani, Sidhant and Tudisco, Francesco and Minervini, Pasquale},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2xbs948bhby9},
  url = {https://doi.org/10.63317/2xbs948bhby9}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R. Lyu. 2021. Towards efficient post-training quantization of pre-trained language models. find
  2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence. find our card
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, and Raul Puri. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374. find our card
  5. Zihan Chen. 2018. Singular value decomposition and its applications in image processing. In Proceedings of the 2018 1st International Conference on Mathematics and Statistics, ICoMS ’18, page 16–22, New York, NY, USA. Association for Computing Machinery. find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HLT (1), pages 2924–2936. Association for Computational Linguistics. find our card
  7. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. find
  8. Rahul Dey and Fathi M Salem. 2017. Gate-variants of gated recurrent unit (GRU) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pages 1597–1600. IEEE. find
  9. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. find
  10. Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. find
  11. Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. Adaptive rank selections for low-rank approximation of language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume… find
  12. Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). find our card
  14. Yen-Chang Hsu, Ting Hua, Sung-En Chang, Qiang Lou, Yilin Shen, and Hongxia Jin. 2022a. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations (ICLR). find
  15. Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022b. Language model compression with weighted low-rank factorization. find
  16. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. find our card
  17. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual difference… find our card
  18. Vimal Kumar, Priyam Srivastava, Ashay Dwivedi, Ishan Budhiraja, Debjani Ghosh, Vikas Goyal, and Ruchika Arora. 2023. Large-language-models (llm)-based ai chatbots: Architecture, in-depth analysis and their performance evaluation. In International Conference on Recent Trends in Image Processing and P… find
  19. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  20. Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. LoSparse: Structured compression of large language models based on low-rank and sparse approximation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma… find
  21. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for llm compression and acceleration. find
  22. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. find
  23. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720. find our card
  24. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. find our card
  25. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. find
  26. Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, and Edoardo M. Ponti. 2024. Dynamic memory compression: Retrofitting llms for accelerated inference. find
  27. Yu Pan, Jing Xu, Maolin Wang, Jinmian Ye, Fei Wang, Kun Bai, and Zenglin Xu. 2019. Compressing recurrent neural networks with tensor ring for action recognition. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press. find
  28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  29. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020a. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. CoRR, abs/2012.15828. find
  30. Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2025. Svd-llm: Truncation-aware singular value decomposition for large language model compression. find
  31. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020b. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find our card
  32. Genta Indra Winata, Andrea Madotto, Jamin Shin, Elham J. Barezi, and Pascale Fung. 2019. On the effectiveness of low-rank matrix factorization for lstm model compression. find
  33. Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, George A. Constantinides, Wayne Luk, and Yiren Zhao. 2025. A3 : an analytical low-rank approximation framework for attention. find
  34. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. find
  35. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2024. Smoothquant: Accurate and efficient post-training quantization for large language models. find
  36. Bosen Yang. 2021. Application of matrix decomposition in machine learning. In 2021 IEEE International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI), pages 133–137. IEEE. find
  37. Yinchong Yang, Denis Krompass, and Volker Tresp. 2017. Tensor-train recurrent neural networks for video classification. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3891–3900. JMLR.org. find
  38. Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2024. Asvd: Activation-aware singular value decomposition for compressing large language models. find
  39. Emanuele Zangrando, Steffen Schotthöfer, Gianluca Ceruti, Jonas Kusch, and Francesco Tudisco. 2023. Rank-adaptive spectral pruning of convolutional layers during training. find
  40. Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. ArXiv, abs/1710.01878. — [arXiv:1710.01878](https://arxiv.org/abs/1710.01878) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sundrani-2026-low-rank-compression-language-models/