Elamur Research⌕
☰
home / papers / 2026 / PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference

PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference

Paper · 2026 · Hao Zhang, Lyu Mengsi, Zhuo Chen et al.

Summary

Production LLM serving often splits inference into two parts: a prefill stage that processes the prompt and builds the KV cache, and a decode stage that generates tokens. These stages usually run on separate nodes, which is called PD disaggregation. Existing pruning methods ignore this split and apply one compression scheme to the whole model, even though the two stages react differently to losing blocks.

PDTrim, from BAAI, is a block-pruning method built around this split:

  • It scores each block's redundancy by comparing the cosine similarity of the block's input and output.
  • The most redundant blocks go directly into a pruning set. Adjacent block pairs that score above a threshold go into a distillation set, where each pair is merged into one block.
  • Both mechanisms remove exactly one block per element, so a temperature-controlled iterative search can swap between candidates to find a better removal configuration.
  • Sensitivity is analysed separately for prefill and decode, so each node type gets its own removal scheme.

On LLaMA and Qwen models, PDTrim matches or beats block-pruning baselines with lower search cost and faster inference. It is useful for compressing models served on disaggregated infrastructure.

Key points

  • PD Unified setting (LLaMA2-13B, 24.4% of parameters pruned): PDTrim averages 47.31 across eight benchmarks, ahead of ShortGPT (44.88), FLAP (44.64), and SLEB (41.13). The dense model scores 55.29.
  • Stage-specific pruning: Using separate prefill and decode schemes raises the LLaMA2-13B average from 47.31 to 49.50.
  • Pruning search cost: PDTrim takes 44.06 s versus 183.6 s for SLEB on LLaMA3.1-8B (five blocks removed), and 96.52 s versus 858.12 s on LLaMA2-13B.
  • Inference latency on LLaMA3.1-8B: At roughly 50% pruning, latency falls from 287.35 ms (dense) to 151.89 ms. This is faster than SliceGPT, LLM-Pruner, and FLAP.
  • Compatibility with quantization: On Qwen2.5-32B with 25% pruning plus 8-bit AWQ, accuracy is nearly unchanged (e.g., PIQA 78.21 vs. 78.45 without quantization). Models around 70B parameters were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Beijing Academy of Artificial IntelligenceBeijing · China32 works · 7,511 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Hao Zhang, Lyu Mengsi, Zhuo Chen, Yulong Ao and Yonghua Lin. 2026. PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1842

@inproceedings{zhang2026pdtrim,
  title = {{PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference}},
  author = {Hao Zhang, Lyu Mengsi, Zhuo Chen, Yulong Ao and Yonghua Lin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1842},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1842}
}

References

57 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2019. Winogrande: An adversarial winograd schema challenge at scale. find
  2. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319. redundant layers to compress large language models. arXiv preprint arXiv:…
  3. Zeqi Chen, Zhaoyang Chu, Yi Gui, Feng Guo, Yao Wan, and Chuan Shi. 2025. Bridging code graphs and large language models for better code understanding. arXiv preprint arXiv:2512.07666. — [arXiv:2512.07666](https://arxiv.org/abs/2512.07666) find
  4. Seonghwan Choi, Beomseok Kang, Dongwon Jo, and Jae-Joon Kim. 2025. Retrospective sparse attention for efficient long-context generation. arXiv preprint arXiv:2508.09001. — [arXiv:2508.09001](https://arxiv.org/abs/2508.09001) find
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  7. G. Ding and 1 others. 2022. Efficient fine-tuning for resource-constrained systems. Proceedings of the Machine Learning Conference. find
  8. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873. find
  9. Xianzhe Dong, Tongxuan Liu, Yuting Zeng, Liangyu Liu, Yang Liu, Siyu Wu, Yu Wu, Hailong Yang, Ke Zhang, and Jing Li. 2025. Hydrainfer: Hybrid disaggregated scheduling for multimodal large language model serving. arXiv preprint arXiv:2505.12658. find
  10. Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024. find
  11. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819. find
  12. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  13. Yuchen Cai, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2302.06557. find
  14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  15. Ruizhe Chen, Yichen Li, Zikai Xiao, and Zuozhu Liu. 2024a. Large language model bias mitigation from the perspective of knowledge editing. arXiv preprint arXiv:2405.09341. — [arXiv:2405.09341](https://arxiv.org/abs/2405.09341) find
  16. Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. 2024b. Streamlining Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of mo…
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  18. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  19. Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and 1 others. 2024a. Memserve: Context caching for disaggregated llm serving with elastic memory pool. arXiv preprint arXiv:2406.17565. — [arXiv:2406.17565](https://arxiv.org/abs/2406.17565) find
  20. Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, and 1 others. 2024b. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181. — [arXiv:2401.11181](https://arxiv.org/abs/2401.11181) find
  21. Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11. — [arXiv:2402.02834](https://arxiv.org/abs/2402.02834) find
  22. C. Li and 1 others. 2023a. Fine-tuning techniques for efficient model adaptation. AI Research Journal. find
  23. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023b. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.
  24. Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, and Zhou Yu. 2025. Injecting adrenaline into llm serving: Boosting resource utilization and throughput via attention disaggregation. arXiv preprint arXiv:2503.20552. — [arXiv:2503.20552](https://arxiv.org/abs/2503.20552) find
  25. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100. find
  26. Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103. find
  27. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.
  28. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.
  29. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Systems, 37:41076–… find
  30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  31. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE. find
  32. A. Qin and 1 others. 2023. Advances in state-of-the-art natural language processing. Journal of NLP Research. find
  33. Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. Mooncake: A kvcache-centric disaggregated architecture for llm serving. arXiv preprint arXiv:2407.00079. find
  34. Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025. — [arXiv:2402.09025](https://arxiv.org/abs/2402.09025) find
  35. Weigao Sun, Jiaxi Hu, Yucheng Zhou, Jusen Du, Disen Lan, Kexin Wang, Tong Zhu, Xiaoye Qu, Yu Zhang, Xiaoyu Mo, and 1 others. 2025. Speed always wins: A survey on efficient architectures for large language models. arXiv preprint arXiv:2508.09834. — [arXiv:2508.09834](https://arxiv.org/abs/2508.09834) find
  36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  37. Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  38. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
  39. Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, and Ming-Chang Yang. 2025. Prefill-decode aggregation or disaggregation? unifying both for goodput-optimized llm serving. arXiv preprint arXiv:2508.01989. find
  40. Tongxi Wang. 2026. Fbs: Modeling native parallel reading inside a transformer. arXiv preprint arXiv:2601.21708.
  41. Thomas Wolf. 2020. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  42. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  43. Yifei Yang, Zouying Cao, and Hai Zhao. 2024b. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187. find
  44. Zhen Yang, Zilun Zhang, Sheng Wang, Jie Li, Meishan Zhang, Zhiyuan Liu, and Maosong Sun. 2021. Knowledge distillation: A survey. arXiv preprint arXiv:2106.05860. — [arXiv:2106.05860](https://arxiv.org/abs/2106.05860) find
  45. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  46. Yuxiao Zhou, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2024. Framequant: Flexible low-bit quantization for transformers. arXiv preprint arXiv:2402.06557. — [arXiv:2402.06557](https://arxiv.org/abs/2402.06557) find
  47. D. Zhang and 1 others. 2023. Parameter-efficient fine-tuning methods for llms. Journal of Machine Learning Research. find
  48. B. Zhu and 1 others. 2023. Large language models: Progress and applications. Advances in NLP. find
  49. Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, and Hao Xu. 2025a. Sensitivity-lora: Low-load sensitivity-based fine-tuning for large language models. arXiv preprint arXiv:2509.09119. find
  50. Hao Zhang, Zhenjia Li, Runfeng Bao, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, Bo Huang, Yuhang Wu, Tianyang Wang, and 1 others. 2025b. Hyperadalora: Accelerating lora rank allocation during training via hypernetworks without sacrificing performance. arXiv preprint arXiv:2510.02630.
  51. Hao Zhang, Mengsi Lyu, Chenrui He, Yulong Ao, and Yonghua Lin. 2025c. Trimtokenator: Towards adaptive visual token pruning for large multimodal models. arXiv preprint arXiv:2509.00320.
  52. Hao Zhang, Mengsi Lyu, Bo Huang, Yulong Ao, and Yonghua Lin. 2025d. Trimtokenator-lc: Towards adaptive visual token pruning for large multimodal models with long contexts. arXiv preprint arXiv:2512.22748. — [arXiv:2512.22748](https://arxiv.org/abs/2512.22748) find
  53. Qifan Zhang, Yunhui Guo, and Yu Xiang. 2024. Continual distillation learning: Knowledge distillation in prompt-based continual learning. Preprint, arXiv:2407.13911. — [arXiv:2407.13911](https://arxiv.org/abs/2407.13911) find
  54. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  55. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210… find
  56. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  57. Yuxiao Zhou, Zhen Wang, Yujun Li, Sheng Wang, Zhiyuan Liu, and Maosong Sun. 2023b. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2302.06557. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-pdtrim-targeted-pruning-prefill-decode/