Elamur Research⌕
☰
home / papers / 2024 / LoRA+: Efficient Low Rank Adaptation of Large Models

LoRA+: Efficient Low Rank Adaptation of Large Models

Paper · 2024 · Soufiane Hayou, Nikhil Ghosh and Bin Yu

Summary

LoRA is a widely used method for cheaply finetuning large pretrained models. It trains two small adapter matrices, A and B, and normally uses one shared learning rate for both. The authors ask whether this shared rate makes sense in wide networks, since A and B have transposed shapes. They model finetuning as width (embedding dimension) grows without bound, starting from a toy linear model and then generalizing. The analysis shows that a single learning rate prevents efficient feature learning at large width.

Their fix, LoRA+, keeps LoRA unchanged except that B's learning rate is a fixed multiple λ (much larger than 1) of A's rate. The method is tested on GPT-2, RoBERTa and Llama-7b. LoRA+ gives roughly 1–2% higher accuracy and up to about 2× faster convergence at no extra compute cost. The gains are largest on harder tasks. Because the change is a one-line hyperparameter adjustment, it is easy to adopt. The paper appeared at ICML 2024.

Key points

  • Core change: the learning rate of B is set to λ·η_A instead of η_B = η_A. This choice is motivated by infinite-width scaling theory, in the spirit of µP.
  • Recommended ratios:
    • RoBERTa with Init[2] (A initialized at zero): λ ≈ 2⁴.
    • Init[1] (B initialized at zero): smaller ratios of about 2²–2³.
    • Llama: about 2¹–2².
  • Roberta-base on MNLI: LoRA+ with η_B = 2⁴·η_A reaches the same test accuracy as standard LoRA in roughly half the training steps.
  • Llama-7b: trained for a single pass over 100k examples drawn from flan-v2 (α = 16, r = 64), it gains about 1.3% MMLU accuracy over the best equal-rate setting. On the easier MNLI task (r = 8), a larger ratio brings little benefit.
  • Limitation: the theory only fixes how the rates scale with width, not the constants. The best λ therefore depends on the model, task and initialization, and still needs to be tuned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations

Also on the byline: Simons Foundation.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Soufiane Hayou, Nikhil Ghosh and Bin Yu. 2024. LoRA+: Efficient Low Rank Adaptation of Large Models. arXiv:2402.12354.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.12354 · arXiv 2402.12354

@misc{hayou2024lora,
  title = {{LoRA+: Efficient Low Rank Adaptation of Large Models}},
  author = {Soufiane Hayou, Nikhil Ghosh and Bin Yu},
  year = {2024},
  doi = {10.48550/arxiv.2402.12354},
  eprint = {2402.12354},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12354}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023. find
  2. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  3. Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=jh-rTtvkGeM. ↗
  4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
  5. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019. find
  6. Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=RbLsYz1Az9. ↗
  7. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  8. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. find
  9. Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning R… find
  10. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  11. Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023.
  12. International Conference on Learning Representations, 2017b. find
  13. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
  14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
  16. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018.
  17. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023.
  18. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023.
  19. Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–50. Springer, 2002. find
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  21. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) find
  22. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022.
  23. Lorenzo Noci, Chuning Li, Mufan Bill Li, Bobby He, Thomas Hofmann, Chris Maddison, and Daniel M. Roy. The shaped transformer: Attention models in the infinite depth-and-width limit, 2023. find
  24. OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023.
  25. G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019. arXiv preprint Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural… find
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9, 2019. find
  27. Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit. arXiv preprint arXiv:2308.01814, 2023. — [arXiv:2308.01814](https://arxiv.org/abs/2308.01814) find
  28. Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation, 2017a. find
  29. Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. In via zero-shot h… find
  30. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023. find
  31. Liu Yang, Steve Hanneke, and Jaime Carbonell. A theory of transfer learning with applications to active learning. Machine learning, 90:161–189, 2013. find
  32. Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513, 2023. — [arXiv:2310.17513](https://arxiv.org/abs/2310.17513) find
  33. A. Proofs In this section, we provide proofs for Proposition 1, Proposition 2, Theorem 1, and some technical details used in the proofs. A.1. Scaling of Neural Networks find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hayou-2024-lora-efficient-low-rank-adaptation/