Elamur Research⌕
☰
home / papers / 2026 / DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning

DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning

Paper · 2026 · Changjun Park, Sejong Yoon and Jaekwang Kim

Summary

Standard LoRA gives every layer the same adapter rank. This spends parameters on layers that matter little and limits the ones that matter most, while hand-tuning per-layer ranks is impractical for models with hundreds of layers. The authors propose DPLoRA, which chooses ranks in two phases. Before training, OPLoRA scores each layer by its mean squared gradient, a cheap approximation of diagonal Fisher information. It models diminishing returns from added rank and solves an Integer Linear Programming problem that assigns one rank per layer under a global parameter budget. During training, a second ILP is re-solved periodically using EMA-smoothed importance computed from the LoRA matrices. Ranks only shrink, following a cubic Bézier budget schedule, and a regularizer discourages abrupt changes. Tests on GLUE with RoBERTa-base and on Alpaca instruction tuning (MT-BENCH) with the LLaMA 3 family show gains over compared PEFT baselines at moderate pruning. Under heavy pruning, accuracy stays competitive while trainable parameters and training time drop sharply. The approach removes manual rank tuning at negligible solver cost.

Key points

  • At pruning rate p = 0.4, full DPLoRA reaches state-of-the-art among the compared PEFT methods on GLUE and cuts parameters by more than 40%.
  • At p = 0.8, compared with standard LoRA, the count of trainable parameters is reduced by more than 80%, and training takes 46% less time.
  • Total ILP solver time across all 11 optimization steps (1 initial plus 10 progressive) is 0.45 s for RoBERTa-base and at most 2.47 s for LLaMA 3.2 3B.
  • Under extreme compression (p = 0.8), the Bézier scheduler beats a linear one on MRPC (89.1 vs 88.0) and RTE (77.6 vs 75.5).
  • Initial layer importance is estimated from five mini-batches using up to 500 samples. Code is available at github.com/cjpark-ai/DPLoRA. The derived rank configurations are task-specific, and the method does not reduce inference latency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sungkyunkwan UniversitySuwon · South Korea9 works · 195 citations

Also on the byline: The College of New Jersey.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Changjun Park, Sejong Yoon and Jaekwang Kim. 2026. DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1693

@inproceedings{park2026dplora,
  title = {{DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning}},
  author = {Changjun Park, Sejong Yoon and Jaekwang Kim},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1693},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1693}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi, Andrew Sabot, Hong Jiang, and HT Kung. 2025. ElaLoRA: Elastic & learnable low-rank adaptation for efficient model fine-tuning. arXiv preprint arXiv:2504.00254. — [arXiv:2504.00254](https://arxiv.org/abs/2504.00254) find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (NAACL): human language techn…
  4. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  5. Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. 2024. LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations. find
  6. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning (ICML), pages 2790–2799. PMLR. find
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations (ICLR).
  8. Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2024. VeRA: Vector-based random matrix adaptation. Proceedings of International Conference on Learning Representations (ICLR).
  9. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  10. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  11. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. DoRA: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML). find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  13. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. find
  14. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. DyLoRA: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL).
  15. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
  16. Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. 2023a. IncreLoRA: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv preprint arXiv:2308.12043. — [arXiv:2308.12043](https://arxiv.org/abs/2308.12043) find
  17. Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. 2023b. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303.
  18. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023c. Adaptive budget allocation for parameter-efficient fine-tuning. International Conference on Learning Representations (ICLR). find
  19. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. AutoLoRA: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Langu… find
  20. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, and Eric Xing. 2023. Judging LLM-as-a-judge with MT-BENCH and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 36:46595–46623.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-dplora-dual-pruning-framework-based/