Elamur Research⌕
☰
home / papers / 2026 / From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models

From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models

Paper · 2026 · Ziyan Wang, Enmao Diao, Qi Le et al.

Summary

Most structured pruning methods for LLMs work locally. They rebuild each layer's outputs separately from calibration data. This keeps perplexity and generic zero-shot behaviour reasonable, but it does not tie the pruning to a downstream task, so task-specific calibration data adds little. The authors return to global pruning and propose GISP (Global Iterative Structured Pruning). It is a post-training method that drops attention heads and MLP channels. Their importance scores are first-order, loss-based estimates, combined per structure and normalized within blocks. Pruning proceeds gradually over a scheduled series of ratios, with no fine-tuning between steps. This avoids the instability of one-shot global pruning and produces nested subnetworks: one pruning run gives models at many sparsity levels ("prune-once, deploy-many"). The loss can be perplexity or a task-oriented margin objective, so pruning can follow a target task. On several Llama, Mistral and reasoning models, GISP lowers perplexity and raises accuracy compared with local baselines, most clearly at high sparsity. This makes compact, deployable LLMs more practical.

Key points

  • Models and benchmarks: Llama2-7B/13B, Llama3-8B and Mistral-0.3-7B were tested on WikiText-2 perplexity and downstream accuracy (CMQA). The largest gains appear at 40–50% sparsity.

  • Reasoning models: On DeepSeek-R1-Distill-Llama-3-8B and Qwen3-8B, task-aligned calibration on GSM8K clearly improves exact-match accuracy. With default C4 calibration, the Wanda-sp baseline drops to 0% at 20% pruning.

  • LoRA fine-tuning: Pruned Llama2-7B models were fine-tuned with LoRA on 2M extra CMQA samples, and GISP stays ahead of Wanda-sp:

    SparsityWanda-spGISP
    20%67.4368.93
    40%58.6660.52

  • Calibration robustness: Cutting the 512K-token CMQA calibration set to 1% shifts accuracy by at most 0.66 points. Resampling with different seeds gives a standard deviation of at most 0.56.

  • Limitations:

    • Gradient-based scoring is costly in memory and compute.
    • Mixture-of-Experts models have not yet been tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IntelSanta Clara · United States15 works · 41,257 citations🎓University of MinnesotaMinneapolis · United States28 works · 2,184 citations🎓University of North Carolina at CharlotteCharlotte · United States6 works · 125 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng and Li Yang. 2026. From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1653

@inproceedings{wang2026local,
  title = {{From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models}},
  author = {Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng and Li Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1653},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1653}
}

References

52 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2023. Fluctuation-based adaptive structured pruning for large language models. Preprint, arXiv:2312.11983. — [arXiv:2312.11983](https://arxiv.org/abs/2312.11983) find
  2. Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. 2024. Sparsellm: Towards global pruning for pre-trained language models. Preprint, arXiv:2402.17946. — [arXiv:2402.17946](https://arxiv.org/abs/2402.17946) find
  3. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
  4. Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. 2020. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129–146. find
  5. Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2019. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791. find
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  10. Rocktim Jyoti Das, Mingjie Sun, Liqun Ma, and Zhiqiang Shen. 2024. Beyond size: How gradients shape pruning decisions in large language models. Preprint, arXiv:2311.04902. — [arXiv:2311.04902](https://arxiv.org/abs/2311.04902) find
  11. DeepSeek-AI, Daya Guo, Dejian Yang, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  12. Enmao Diao, Ganghua Wang, Jiawei Zhan, Yuhong Yang, Jie Ding, and Vahid Tarokh. 2023. Pruning deep neural networks from a sparsity perspective. Preprint, arXiv:2302.05601. — [arXiv:2302.05601](https://arxiv.org/abs/2302.05601) find
  13. Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. 2024. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. Preprint, arXiv:2406.02924. — [arXiv:2406.02924](https://arxiv.org/abs/2406.02924) find
  14. Xin Dong, Shangyu Chen, and Sinno Jialin Pan. 2017. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Preprint, arXiv:1705.07565. — [arXiv:1705.07565](https://arxiv.org/abs/1705.07565) find
  15. Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. find
  16. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. Preprint, arXiv:2301.00774. find
  17. Aaron Grattafiori, Abhimanyu Dubey, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  18. Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. Preprint, arXiv:1510.00149. find
  19. Babak Hassibi and David Stork. 1992. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmann. find
  20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Preprint, arXiv:2009.13081.
  22. Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened llama: Depth pruning for large language models with comparison of retraining methods. Preprint, arXiv:2402.02834. — [arXiv:2402.02834](https://arxiv.org/abs/2402.02834) find
  23. Eldar Kurtic, Elias Frantar, and Dan Alistarh. 2023. Ziplm: Inference-aware structured pruning of language models. Preprint, arXiv:2302.04089. — [arXiv:2302.04089](https://arxiv.org/abs/2302.04089) find
  24. Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann. find
  25. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  26. Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. Preprint, arXiv:1711.05769. find
  27. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. Preprint, arXiv:2403.03853.
  28. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641. find
  30. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. 2025. Stop overthinking: A survey on efficient reasoning for large language models. Preprint, arXiv:2503.16419.
  31. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. Preprint, arXiv:2306.11695. find
  32. Shengkun Tang, Oliver Sieberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. 2025. Darwinlm: Evolutionary structured pruning of large language models. Preprint, arXiv:2502.07780. — [arXiv:2502.07780](https://arxiv.org/abs/2502.07780) find
  33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  34. Hugo Touvron and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  36. Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. 2024. Efficient large language models: A survey. Preprint, arXiv:2312.03863.
  37. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. find
  38. Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. 2024. Model compression and efficient inference for large language models: A survey. Preprint, arXiv:2402.09748. — [arXiv:2402.09748](https://arxiv.org/abs/2402.09748)
  39. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. Preprint, arXiv:1906.10771. find
  40. Zi Wang, Chengcheng Li, and Xiangyang Wang. 2021. Convolutional neural network pruning with structural redundancy reduction. Preprint, arXiv:2104.03438. — [arXiv:2104.03438](https://arxiv.org/abs/2104.03438) find
  41. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) find
  42. BigScience Workshop, :, Teven Le Scao, Angela Fan, and 1 others. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. find
  43. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. Preprint, arXiv:1611.06440. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) find
  44. An Yang, Anfeng Li, Baosong Yang, and Others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  45. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and 1 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  46. Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. 2023. Global vision transformer pruning with hessian-aware saliency. Preprint, arXiv:2110.04869. — [arXiv:2110.04869](https://arxiv.org/abs/2110.04869) find
  47. Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FitzGerald, and Abhishek Kumar. 2025b. Wanda++: Pruning large language models via regional… find
  48. Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. 2025. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. Preprint, arXiv:2310… — [arXiv:2310.05175](https://arxiv.org/abs/2310.05175) find
  49. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  50. Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2024. Loraprune: Structured pruning meets low-rank parameter-efficient fine-tuning. Preprint, arXiv:2305.18403. find
  51. Nan Zhang, Eugene Kwek, Yusen Zhang, Ngoc-Hieu Nguyen, Prasenjit Mitra, and Rui Zhang. 2025. When reasoning meets compression: Understanding the effects of llms compression on large reasoning models. Preprint, arXiv:2504.02010. — [arXiv:2504.02010](https://arxiv.org/abs/2504.02010) find
  52. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. A survey on efficient inference for large language models. Preprint, arXiv:2404.14294. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-local-global-revisiting-structured-pruning/