Elamur Research⌕
☰
home / papers / 2025 / LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation

LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation

Paper · 2025 · Hongyun Zhou, Xiangyu Lu, Wang Xu et al.

Summary

LoRA lowers the cost of fine-tuning. For billion-parameter models, though, its trainable weights still need considerable compute. Storage also adds up when providers keep a separate adapter for each user. Earlier pruning methods (Sparse Adapter, AdaLoRA, SoRA) rank adapter importance using only properties of the weights ΔW. The authors argue that what actually alters the frozen model is the adapter's output ΔWx, which depends on both the weights and the task data.

A pilot study with RoBERTa-base on RTE and MRPC found that per-layer output norms are tightly clustered. Some layers contribute almost nothing, and which layers matter differs by task. Building on this, LoRA-drop works in three steps:

  • It briefly trains LoRA on a stratified sample of the training data.
  • It scores each layer by its normalized summed squared output norm.
  • It keeps dedicated LoRA only for the highest-scoring layers, up to a cumulative importance threshold. All other layers share a single LoRA, and the model is then fine-tuned under this reduced setup.

Across NLU and NLG benchmarks with three models of different sizes, LoRA-drop performs on par with standard LoRA and full fine-tuning. It does so with roughly half of the LoRA parameters, offering a cheap, data-aware way to shrink adapters.

Key points

  • Models tested: RoBERTa-base, RoBERTa-large and Llama2-7b. Benchmarks: GLUE for NLU; E2E, DART, DialogSum and GSM8K for NLG (Llama2-7b only), scored with BLEU, ROUGE and accuracy.
  • Default hyperparameters: the sampling ratio α is 10% of the training data, and the cumulative importance threshold T is 0.9, which keeps about half of the layers' LoRA on average.
  • Baselines: FULL-FT, LoRA, Sparse Adapter, VeRA, Tied-LoRA and SoRA.
  • Ablations: removing low-importance LoRA outright instead of sharing it costs about 0.7 points on average. A fixed top-k layer selection trails LoRA-drop by about 0.5 points, and the inverse-importance variant does slightly worse than random selection.
  • Stated limitations: the method works at the coarse level of whole LoRA modules, and it does not reduce inference cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao and Muyun Yang. 2025. LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhou2025lora,
  title = {{LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation}},
  author = {Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao and Muyun Yang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.371/}
}

References

32 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics, pages 5062–5074.
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  3. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Proceedings of the Conference on Neural Information Processing Systems.
  4. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  5. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, pages 123– 156. find
  6. Anchun Gui and Han Xiao. 2023. Hifi: High-information attention heads hold for parameter-efficient model adaptation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8521–8537. find
  7. Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff pruning. In Proceedings of the annual Meeting of the Association for Computational Linguistics.
  8. Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608.
  9. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. In Proceedings of the International Conference on Learning Representations.
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034. find
  11. Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. In Findings of the Association for Computational Linguistics, pages 2184–2190. find
  12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning, pages 2790–2799. find
  13. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations.
  14. Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2024. VeRA: Vector-based random matrix adaptation. In Proceedings of the Annual Meeting of the International Conference on Learning Representations.
  15. Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090. — [arXiv:1911.03090](https://arxiv.org/abs/1911.03090) find
  16. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S Torr. 2018. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340. — [arXiv:1810.02340](https://arxiv.org/abs/1810.02340) find
  17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
  18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundat…
  19. Joint Conference on Natural Language Processing, pages 4582–4597. find
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  21. Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. Dora: Weight-decomposed low-rank adaptation. In Proceedings of Forty-first International Conference on Machine Learning. find
  22. Wei Liu, Ying Qin, Zhiyuan Peng, and Tan Lee. 2024b. Sparsely shared lora on whisper for child speech recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11751–11755. IEEE. find
  23. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385.
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations.
  26. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1–9.
  27. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2022. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the International Conference on Learning Representations. find
  28. Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, C… — DART: Open-Domain Structured Data Record to Text Generation
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the annual meeting of the Association for Computational Linguistics, pages 311–318.
  30. Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. 2024. Tied-lora: Enhancing parameter efficiency of lora with weight tying. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape… find
  31. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 7930–7946.
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2025-lora-drop-efficient-lora-parameter/