Elamur Research⌕
☰
home / papers / 2026 / Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration

Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration

Paper · 2026 · Guangxin Wu, Hao Zhang, Zhang Zhibin et al.

Summary

Large language models are expensive to deploy because of their memory use, compute cost and latency. Unstructured pruning creates irregular sparsity that standard hardware cannot exploit well. The authors therefore focus on structured pruning, which removes whole channels and attention heads. They build on FLAP, which scores channels by activation variance weighted by weight magnitude, prunes in one pass and offsets the removed output with a bias term. They add two components:

  1. Mixed-domain calibration set. It combines Wikipedia, Common Crawl, source code and mathematical text, so that activation statistics are not skewed toward a single domain.
  2. Iterative calibration. Statistics are recomputed after each pruning round, because earlier removals change downstream activations.

Experiments on the Qwen2.5 family compare the method against Wanda-sp and FLAP. The paper reports better accuracy retention at equal compression, although the detailed baseline comparison was not in the excerpt provided. Gradual pruning matters most at high sparsity. The approach needs no retraining and produces models that run on ordinary hardware.

Key points

  • Setup: Baselines are Wanda-sp and FLAP. Models are Qwen2.5-7B, 14B and 32B. Evaluation uses zero-shot ARC-c, ARC-e, HellaSwag, OBQA, PIQA and Winogrande, plus WikiText2 perplexity.

  • Iterative pruning results:

    • At 25% pruning, perplexity barely changes with the number of steps.
    • At 50% pruning, most of the gain comes in the first 3–4 iterations, and 4–6 steps are usually enough.
    • For Qwen2.5-14B at 50%, one-shot pruning degrades the model severely, while six steps bring perplexity down to roughly 44.

  • Mixed-domain calibration on Qwen2.5-14B (average zero-shot accuracy; the dense model scores 65.42):

    Pruning ratioSingle-domainMixed-domain
    25%53.9653.94
    50%35.9037.35

    The benefit therefore shows up mainly at high sparsity; at 25% the two settings are essentially tied.

  • Method details: Low-fluctuation channels are removed, and a bias computed from mean activations compensates for them. Statistics from the different domains are combined as a weighted mixture.

  • Limitation: Because of compute limits, the method was not tested on models of around 70B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Computing TechnologyBeijing · China51 works · 3,411 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Guangxin Wu, Hao Zhang, Zhang Zhibin, Jiafeng Guo and Xueqi Cheng. 2026. Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.1

@inproceedings{wu2026iterative,
  title = {{Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration}},
  author = {Guangxin Wu, Hao Zhang, Zhang Zhibin, Jiafeng Guo and Xueqi Cheng},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.1},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.1}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873. find
  2. C. Li and 1 others. 2023a. Fine-tuning techniques for efficient model adaptation. AI Research Journal. find
  3. Yong Li, Wei Du, Liquan Han, Zhenjian Zhang, and Tongtong Liu. 2023b. A communication-efficient, privacy-preserving federated learning algorithm based on two-stage gradient pruning and differentiated differential privacy. Sensors, 23(23):9305. find
  4. Anonymous. 2024. Unstructured pruning and low rank factorisation of self-supervised pre-trained speech models. IEEE Transactions on Audio, Speech, and Language Processing, pages 1046–1058. find
  5. Sheng Liao and 1 others. 2023. Can unstructured pruning reduce the depth in deep neural networks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. find
  6. Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024. find
  7. Yang Luo, Shiru Wang, Jun Liu, Jiaxuan Xiao, Rundong Xue, Zeyu Zhang, Hao Zhang, Yu Lu, Yang Zhao, and Yutong Xie. 2025. Pathohr: Breast cancer survival prediction on high-resolution pathological images. arXiv preprint arXiv:2503.17970. — [arXiv:2503.17970](https://arxiv.org/abs/2503.17970) find
  8. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  9. Chenrui Ma, Rongchang Zhao, Xi Xiao, Hongyang Xie, Tianyang Wang, Xiao Wang, Hao Zhang, and Yanning Shen. 2025. Cad-vae: Leveraging correlation-aware latents for comprehensive fair disentanglement. arXiv preprint arXiv:2503.07938. — [arXiv:2503.07938](https://arxiv.org/abs/2503.07938) find
  10. Yuchen Cai, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2302.06557. find
  11. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  13. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. find
  14. Zhaoyang Cong, Ziyang Wang, Hao Zhang, Guowei Zheng, Keming Cao, Lina Zhao, Ruipeng Song, Jianqing Li, and Chengyu Liu. 2025. Hierarchical multi-scale feature fusion network for multi-center major depressive disorder classification with t1-weighted mri. In Annual International Conference of the IEEE… find
  15. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  16. G. Ding and 1 others. 2022. Efficient fine-tuning for resource-constrained systems. Proceedings of the Machine Learning Conference. find
  17. Xuyin Qi, Zeyu Zhang, Canxuan Gang, Hao Zhang, Lei Zhang, Zhiwei Zhang, and Yang Zhao. 2025a. find
  18. Mediaug: Exploring visual augmentation in medical imaging. In Annual Conference on Medical Image Understanding and Analysis, pages 218–232. Springer. find
  19. Heng Zhang, Tianyi Zhang, Yuling Shi, Xiaodong Gu, Yaomin Shen, Zijian Zhang, Yilei Yuan, Hao Zhang, and Jin Huang. 2025b. Can representation gaps be the key to enhancing robustness in graph-text alignment? arXiv preprint arXiv:2510.12087. — [arXiv:2510.12087](https://arxiv.org/abs/2510.12087) find
  20. Xuyin Qi, Zeyu Zhang, Huazhan Zheng, Mingxi Chen, Numan Kutaiba, Ruth Lim, Cherie Chiang, Zi En Tham, Xuan Ren, Wenxin Zhang, and 1 others. 2025b. Medconv: Convolutions beat transformers on long-tailed bone density prediction. IJCNN2025. find
  21. Qifan Zhang, Yunhui Guo, and Yu Xiang. 2024. Continual distillation learning: Knowledge distillation in prompt-based continual learning. Preprint, arXiv:2407.13911. — [arXiv:2407.13911](https://arxiv.org/abs/2407.13911) find
  22. A. Qin and 1 others. 2023. Advances in state-of-the-art natural language processing. Journal of NLP Research. find
  23. Heng Zheng, Yuling Shi, Xiaodong Gu, Haochen You, Zijian Zhang, Lubin Gan, Hao Zhang, Wenjun Huang, and Jin Huang. 2025a. Graphgeo: Multi-agent debate framework for visual geo-localization with heterogeneous graph neural networks. arXiv preprint arXiv:2511.00908. — [arXiv:2511.00908](https://arxiv.org/abs/2511.00908) find
  24. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  25. Heng Zheng, Haochen You, Zijun Liu, Zijian Zhang, Lubin Gan, Hao Zhang, Wenjun Huang, and Jin Huang. 2025b. G2rammar: Bilingual grammar modeling for enhanced text-attributed graph learning. arXiv preprint arXiv:2511.00911. — [arXiv:2511.00911](https://arxiv.org/abs/2511.00911) find
  26. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. find
  27. Yuxiao Zhou, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2302.06557. find
  28. F. Wang and 1 others. 2023. Practical applications of llms in specialized domains. Specialized AI Applications. find
  29. Thomas Wolf. 2020. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  30. Yuxiao Zhou, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2024. Framequant: Flexible low-bit quantization for transformers. arXiv preprint arXiv:2402.06557. — [arXiv:2402.06557](https://arxiv.org/abs/2402.06557) find
  31. Yu-Hang Wu, Yu-Jie Xiong, Hao Zhang, Jia-Chen Zhang, and Zheng Zhou. 2025. Sugar-coated poison: Benign generation unlocks llm jailbreaking. find
  32. B. Zhu and 1 others. 2023. Large language models: Progress and applications. Advances in NLP. find
  33. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  34. Zhen Yang, Zilun Zhang, Sheng Wang, Jie Li, Meishan Zhang, Zhiyuan Liu, and Maosong Sun. 2021. Knowledge distillation: A survey. arXiv preprint arXiv:2106.05860. — [arXiv:2106.05860](https://arxiv.org/abs/2106.05860) find
  35. Zhengwu Yang and Han Zhang. 2022. Comparative analysis of structured pruning and unstructured pruning. In Frontier Computing, page 112. Springer. find
  36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  37. D. Zhang and 1 others. 2023. Parameter-efficient fine-tuning methods for llms. Journal of Machine Learning Research. find
  38. Heng Zhang, Haichuan Hu, Yaomin Shen, Weihao Yu, Yilei Yuan, Haochen You, Guo Cheng, Zijian Zhang, Lubin Gan, Huihui Wei, and 1 others. 2025a. Asy-moe: Leveraging modal asymmetry for enhanced expert specialization in large vision-language models. arXiv preprint arXiv:2509.12715. — [arXiv:2509.12715](https://arxiv.org/abs/2509.12715) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-iterative-structured-pruning-large-language/