Elamur Research⌕
☰
home / papers / 2026 / LaCo: Layer-wise Compensation for Pruned Large Language Models

LaCo: Layer-wise Compensation for Pruned Large Language Models

Paper · 2026 · Yingen Liu, Fan Wu, Panxuyan et al.

Summary

Pruning makes large language models cheaper to run, but it often damages their quality. Common recovery methods such as LoRA fine-tune the whole model on task data. They do not target the actual cause of the damage. The authors give a Lipschitz-based bound showing that small per-layer errors from pruning add up and grow as they pass through later layers.

Their method, LaCo, fixes the model one layer at a time. For each layer, it keeps the pruning mask fixed and adjusts only the remaining weights so the layer's hidden states match those of the original dense model under an MSE objective. The corrected output then feeds the next layer, which stops errors from spreading. The method works with unstructured (Wanda), structured, and semi-structured (SparseGPT, N:M) pruning.

Across Qwen-2.5-7B and Llama-2 models, LaCo generally beats LoRA, especially at 70% sparsity. It also needs far less GPU memory and calibration data. This makes it practical to recover pruned models on consumer GPUs and in domains where data is scarce.

Key points

  • With Wanda at 70% sparsity on Llama-2-13B, LaCo lowered mean perplexity to 51.94, versus 71.86 for LoRA and 235.15 with no recovery. It raised mean zero-shot accuracy to 57.77, versus 54.27 for LoRA.
  • On LLaMA-2-7B (Wanda, 70%), peak memory was 6.2 GB for LaCo versus 43.7 GB for LoRA, about a 7× reduction. LaCo ran longer, at 6h 50m versus 4h 20m.
  • LaCo uses 2,048 unlabeled calibration samples, while LoRA uses about 50k supervised examples, roughly 25× less data.
  • In the ablation, compensating both attention and MLP gave WikiText2 perplexity of 27.25. Attention alone gave 38.36 and MLP alone gave 43.19.
  • The default settings are a learning rate of 5×10⁻⁵ and 10 epochs. Performance levels off once the calibration set reaches a few thousand samples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hunan UniversityChangsha · China4 works · 304 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yingen Liu, Fan Wu, Panxuyan, Ruihui Li, Zhuo Tang and Kenli Li. 2026. LaCo: Layer-wise Compensation for Pruned Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1342

@inproceedings{liu2026laco,
  title = {{LaCo: Layer-wise Compensation for Pruned Large Language Models}},
  author = {Yingen Liu, Fan Wu, Panxuyan, Ruihui Li, Zhuo Tang and Kenli Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1342},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1342}
}

References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873. find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  3. Hao Chen, Haoze Li, Zhiqing Xiao, Lirong Gao, Qi Zhang, Xiaomeng Hu, Ningtao Wang, Xing Fu, and Junbo Zhao. 2025a. Alps: Attention localization and pruning strategy for efficient adaptation of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11764–… find
  4. Xiaodong Chen, Yuxuan Hu, Xiaokang Zhang, Yanling Wang, Cuiping Li, Hong Chen, and Jing Zhang. 2025b. P 2 law: Scaling law for post-training after model pruning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5668–5686, Vienn… find
  5. Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2024. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer. find
  9. Xuan Ding, Rui Sun, Yunjian Zhang, Xiu Yan, Yueqi Zhou, Kaihao Huang, Suzhong Fu, Angelica I. Aviles-Rivero, Chuanlong Xie, and Yao Zhu. 2025. A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs. Preprint, arXiv:2502.19159. — [arXiv:2502.19159](https://arxiv.org/abs/2502.19159) find
  10. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR. find
  11. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  12. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819. find
  13. Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330.
  14. Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. 2025. SlimLLM: Accurate Structured Pruning for Large Language Models. Preprint, arXiv:2505.22689. — [arXiv:2505.22689](https://arxiv.org/abs/2505.22689) find
  15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  16. Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. 2021. Stable resnet. In International Conference on Artificial Intelligence and Statistics, pages 1324– 1332. PMLR. find
  17. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  19. Hyunjik Kim, George Papamakarios, and Andriy Mnih. 2021. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562–5571. PMLR. find
  20. Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, and Jie Tang. 2025. A survey of post-training scaling in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Comput…
  21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  22. Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. 2023. Lipsformer: Introducing lipschitz continuity to vision transformers. arXiv preprint arXiv:2304.09856. — [arXiv:2304.09856](https://arxiv.org/abs/2304.09856) find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  24. Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2025. MoDeGPT: Modular Decomposition for Large Language Model Compression. Preprint, arXiv:2408.09632. find
  25. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740. find
  26. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100. find
  27. Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmiatis, Nikos Komodakis, and Sergey Zagoruyko. 2025. ReplaceMe: Network Simplification via Layer Pruning and Linear Transformations. Preprint, arXiv:2505.02819. — [arXiv:2505.02819](https://arxiv.org/abs/2505.02819) find
  28. Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Xi Chen, Cunhang Fan, Zhao Lv, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging. Preprint, arXiv:2406.16330. find
  29. Songtao Liu and Peng Liu. 2025. High-Layer Attention Pruning with Rescaling. Preprint, arXiv:2507.01900. — [arXiv:2507.01900](https://arxiv.org/abs/2507.01900) find
  30. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. find
  31. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  33. Pingjie Wang, Ziqing Fan, Shengchao Hu, Zhe Chen, Yanfeng Wang, and Yu Wang. 2025. Reconstruct the Pruned Model Without Retraining. IEEE Journal of Selected Topics in Signal Processing, pages 1–12. find
  34. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Advances in Neural Information Processing Systems, volume 33, pages 5776–5788. Curran Associates, Inc. find
  35. Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019. Understanding and improving layer normalization. Advances in neural information processing systems, 32. find
  36. Qwen: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. Qwen2.5 technical report. Preprint, arX… find
  37. Hao Yu and Jianxin Wu. 2023. Compressing Transformers: Features Are Low-Rank, but Weights Are Not! Proceedings of the AAAI Conference on Artificial Intelligence, 37(9):11007–11015. find
  38. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  39. Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2024. Loraprune: Structured pruning meets low-rank parameter-efficient fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3013–3026. find
  40. Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2025. BlockPruner: Fine-grained pruning for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5065–5080, Vienna, Austria. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-laco-layer-wise-compensation-pruned/