Elamur Research⌕
☰
home / papers / 2026 / HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance

HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance

Paper · 2026 · Hao Zhang, Zhenjia Li, Yifan Gao et al.

Summary

AdaLoRA improves on LoRA by giving different weight matrices different ranks. It does this by expressing each update in SVD form (P, Λ, Q) and pruning unimportant singular values. The cost is slow convergence and heavy computation during training. HyperAdaLoRA addresses this. Instead of updating P, Λ and Q directly through backpropagation, it trains hypernetworks that take the current values of these components and output updated ones. One shared hypernetwork serves each component type across all layers. The hypernetworks are built from a BERT attention layer, so they can model dependencies among parameters. Rank allocation still works by pruning, now applied to the output of the hypernetwork that produces the singular values.

The authors test the method on:

  • NLU tasks with DeBERTa-v3-base, including RTE and WNLI;
  • generation tasks with LLaMA3.1-8B, Qwen2.5-7B and Qwen2.5-14B, on Stanford Alpaca, Magpie-Pro-300K-Filtered and OpenPlatypus.

HyperAdaLoRA reaches convergence faster, and its final quality matches or exceeds AdaLoRA's. The approach also carries over to SRD and rank-sampling LoRA variants. This matters because adaptive-rank fine-tuning becomes cheaper to run.

Key points

  • Total training time drops consistently compared with AdaLoRA:
    • LLaMA3.1-8B: 8125 → 6650 s on Stanford Alpaca, 19,600 → 15,720 s on Magpie-Pro-300K-Filtered, 11,900 → 9750 s on OpenPlatypus.
    • Qwen2.5-7B: 4240 → 3500 s, 15,000 → 11,000 s and 6750 → 5500 s on the same three datasets.
  • Per-step cost is also slightly lower across batch sizes 1–64. At batch size 1, memory falls from 2758 to 2722 MB and latency from 123.16 to 118.10 ms per step.
  • In the ablation, three hypernetwork designs were compared on DeBERTa-v3-base. The BERT attention layer converged fastest, CNN came second, and MLP was slowest.
  • Results are stable across values of the hyperparameter γ, as shown with LLaMA3.1-8B on Alpaca and Magpie.
  • As a limitation, the authors did not test the method on very large models, such as those with 70B parameters, because of limited compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Latent semantic indexing · 29 works

Cite

Cite as Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang and Hao Xu. 2026. HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1648

@inproceedings{zhang2026hyperadalora,
  title = {{HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance}},
  author = {Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang and Hao Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1648},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1648}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  2. Zeqi Chen, Zhaoyang Chu, Yi Gui, Feng Guo, Yao Wan, and Chuan Shi. 2025. Bridging code graphs and large language models for better code understanding. arXiv preprint arXiv:2512.07666. — [arXiv:2512.07666](https://arxiv.org/abs/2512.07666) find
  3. Seonghwan Choi, Beomseok Kang, Dongwon Jo, and Jae-Joon Kim. 2025. Retrospective sparse attention for efficient long-context generation. arXiv preprint arXiv:2508.09001. — [arXiv:2508.09001](https://arxiv.org/abs/2508.09001) find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Chongyuan Dai, Yaling Shen, Jinpeng Hu, Zihan Gao, Jia Li, Yishun Jiang, Yaxiong Wang, Liu Liu, and Zongyuan Ge. 2026. Tears or cheers? benchmarking llms via culturally elicited distinct affective responses. arXiv preprint arXiv:2601.13024. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  7. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. arXiv preprint arXiv:1609.09106. find
  10. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing (2021). URL https://arxiv.org/abs/2111.09543. ↗
  11. N. Houlsby, A. Giurgiu, S. Jastrzebski, and 1 others. 2019. Parameter-efficient transfer learning for nlp. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019:279–285. find
  12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  13. Yahao Hu, Yifei Xie, Tianfeng Wang, Man Chen, and Zhisong Pan. 2023. Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics, 11(20):4317. find
  14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  15. Louis Kirsch, Julius Kunze, and David Barber. 2018. Modular networks: Learning to decompose neural computation. Advances in neural information processing systems, 31. find
  16. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, cheap, and powerful refinement of llms. find
  17. B. Lester, R. Al-Rfou, and N. Constant. 2021. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021:3045–3061.
  18. Mengtian Li, Jinshu Chen, Wanquan Feng, Bingchuan Li, Fei Dai, Songtao Zhao, and Qian He. 2025. Hyperlora: Parameter-efficient adaptive generation for portrait synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13114–13123. find
  19. X. Li and P. Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation tasks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021:4582–4597. find
  20. Chen Liang, Nikos Karampatziakis, Tuo Zhao, and Weizhu Chen. 2022. Hart: Efficient adaptation via regularized autoregressive parameter generation. arXiv preprint arXiv:2207.01411. — [arXiv:2207.01411](https://arxiv.org/abs/2207.01411) find
  21. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. find
  22. Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364.
  23. Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. 2024b. Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403.16187. find
  24. Jonathan Lorraine and David Duvenaud. 2018. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv:1802.09419. — [arXiv:1802.09419](https://arxiv.org/abs/1802.09419) find
  25. Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. 2024. Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. arXiv preprint arXiv:2405.17357. find
  26. Jesus-German Ortiz-Barajas, Helena Gomez-Adorno, and Thamar Solorio. 2024. Hyperloader: Integrating hypernetwork-based lora and adapter layers into multi-task transformers for sequence labelling. arXiv preprint arXiv:2407.01411. — [arXiv:2407.01411](https://arxiv.org/abs/2407.01411) find
  27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  28. Haoxiang Shi, Rongsheng Zhang, Jiaan Wang, Cen Wang, Yinhe Zheng, and Tetsuya Sakai. 2022. Layer-connect: Hypernetwork-assisted inter-layer connector to enhance parameter efficiency. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3120–3126. find
  29. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  30. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2022. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558.
  31. Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  32. Tongxi Wang. 2026. Fbs: Modeling native parallel reading inside a transformer. arXiv preprint arXiv:2601.21708.
  33. Thomas Wolf. 2020. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. synthesis from scratch by prompting aligned llms with nothing. ArXiv, abs/2406.08464.
  34. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  35. Shufan Yang, Zifeng Cheng, Zhiwei Jiang, Yafeng Yin, Cong Wang, Shiping Ge, Yuchen Fu, and Qing Gu. 2026. Regionmarker: A region-triggered semantic watermarking framework for embedding-as-a-service copyright protection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pag… find
  36. E. Zaken, Y. Goldberg, and S. Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformers. Transactions of the Association for Computational Linguistics (TACL), 10:1–16. find
  37. Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. 2023a. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv preprint arXiv:2308.12043. — [arXiv:2308.12043](https://arxiv.org/abs/2308.12043) find
  38. Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, and Hao Xu. 2025a. Sensitivity-lora: Low-load sensitivity-based fine-tuning for large language models. arXiv preprint arXiv:2509.09119. find
  39. Hao Zhang, Mengsi Lyu, Zhuo Chen, Xingrun Xing, Yulong Ao, and Yonghua Lin. 2025b. Pdtrim: Targeted pruning for prefill-decode disaggregation in inference. arXiv preprint arXiv:2509.04467.
  40. Hao Zhang, Mengsi Lyu, Chenrui He, Yulong Ao, and Yonghua Lin. 2025c. Trimtokenator: Towards adaptive visual token pruning for large multimodal models. arXiv preprint arXiv:2509.00320.
  41. Hao Zhang, Mengsi Lyu, Bo Huang, Yulong Ao, and Yonghua Lin. 2025d. Trimtokenator-lc: Towards adaptive visual token pruning for large multimodal models with long contexts. arXiv preprint arXiv:2512.22748. — [arXiv:2512.22748](https://arxiv.org/abs/2512.22748) find
  42. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.
  43. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. arXiv preprint arXiv:2403.09113. find
  44. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-hyperadalora-accelerating-lora-rank-allocation/