Elamur Research⌕
☰
home / papers / 2026 / G-LoRA: Global-Local Decoupled Low-Rank Adaptation

G-LoRA: Global-Local Decoupled Low-Rank Adaptation

Paper · 2026 · Jiahao Xiong, Yihong Huang, Yihe Liu et al.

Summary

LoRA and adaptive variants such as AdaLoRA give a single rank to an entire weight update matrix ∆W. This ignores the fact that parameter importance varies inside one matrix. Sparse fine-tuning does capture that variation, but it needs a memory-heavy gradient warm-up. G-LoRA addresses the gap by splitting each update into two parts. A low-rank G-Adapter spans the whole matrix. An L-Adapter gets a larger share of the rank and acts on a sub-block where the most influential parameters have been grouped. The grouping comes from permuting rows and columns by a first-order Taylor importance estimate. During training, this local region is repeatedly shrunk and re-initialized with randomized SVD, which concentrates capacity further. The authors derive upper and lower bounds on the attainable rank, showing that it exceeds LoRA's under an equal parameter budget. Across benchmarks testing commonsense and mathematical reasoning as well as code writing, G-LoRA beats LoRA, DoRA, PiSSA, RaSA, RoSA and GraLoRA by as much as 2.7 accuracy points. This makes it a practical way to extract more from a fixed fine-tuning budget at small extra cost.

Key points

  • With the L-Adapter restricted to 1/16 of the matrix, theory gives roughly 1.5–2.5× higher effective rank than LoRA at the same budget. The gain grows as the local region gets smaller.
  • On LLaMA2-7B at rank 8, the Entropy-Based Effective Rank (EBER) of the updates is 137.23 for G-LoRA. LoRA reaches 5.57 and GraLoRA 13.14.
  • Training time rises only about 8% over LoRA, mostly from the reordering steps.
  • Even with re-initialization switched off, G-LoRA scores 70.4% versus LoRA's 70.0%. Re-initializing every 2–4 epochs works best.
  • Initializing the G-Adapter with small singular values and the L-Adapter with large ones gave the top average of 71.8%. Limitations: experiments cover English text only, and the global/local rank ratio is set by hand.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓East China Normal UniversityShanghai · China69 works · 2,490 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao and Kai Zhang. 2026. G-LoRA: Global-Local Decoupled Low-Rank Adaptation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1005

@inproceedings{xiong2026g,
  title = {{G-LoRA: Global-Local Decoupled Low-Rank Adaptation}},
  author = {Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao and Kai Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1005},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1005}
}

References

54 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  2. Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. 2024. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405. — [arXiv:2401.16405](https://arxiv.org/abs/2401.16405) find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  4. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1–9. Association for Computational Linguistics (ACL).
  5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
  6. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca. ↗
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code.
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  9. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  11. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  12. Haonan Dong, Wenhao Zhu, Guojie Song, and Liang Wang. 2025. Aurora: Breaking low-rank bottleneck of lora with nonlinear mapping. In Advances in Neural Information Processing Systems. find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  14. Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. 2011. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217– 288. find
  15. Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy, and Guillaume Rabusseau. 2024. Rosa: Random subspace adaptation for efficient fine-tuning. Preprint, arXiv:2407.07802. — [arXiv:2407.07802](https://arxiv.org/abs/2407.07802) find
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026–1034. find
  17. Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. 2025. RaSA: Rank-sharing low-rank adaptation. In Proceedings of the 13th International Conference on Learning Representations (ICLR). find
  18. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523–533.
  19. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 2790–2799. PMLR. find
  20. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR).
  21. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (…
  22. Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. 2025a. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations. find
  23. Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Liuyang, Jing Lin, Yiwu Yao, and Rongrong Ji. 2025b. Dynamic low-rank sparse adaptation for large language models. In The Thirteenth International Conference on Learning Representations. find
  24. Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, and Eunhyeok Park. 2025. Gralora: Granular low-rank adaptation for parameter-efficient fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  25. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. find
  26. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of NAACL, pages 1152–1157. find
  27. Wen Lai, Alexander Fraser, and Ivan Titov. 2025. Joint localization and activation editing for low-resource fine-tuning. In Forty-second International Conference on Machine Learning. find
  28. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059.
  29. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. 2025a. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In Advances in Neural Information Processing Systems. find
  30. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. 2025b. Bora: Towards more expressive low-rank adaptation with block diversity. Preprint, arXiv:2508.06953. — [arXiv:2508.06953](https://arxiv.org/abs/2508.06953) find
  31. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582–4597. find
  32. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. find
  33. Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. 2024. Svft: Parameter-efficient fine-tuning with singular vectors. volume 37, pages 41425–41446. find
  34. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML). find
  35. Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, and Aaron Xuxiang Tian. 2024b. Para: Parameter-efficient fine-tuning with prompt-aware representation adjustment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 728–737. find
  36. Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. 2025. LIFT the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. In Forty-second International Conference on Machine Learning. find
  37. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 121038–121072.
  38. Daiye Miao, Yufang Liu, Jie Wang, Changzhi Sun, Yunke Zhang, Demei Yan, Shaokang Dong, Qi Zhang, and Yuanbin Wu. 2025. TASO: Task-aligned sparse optimization for parameter-efficient model adaptation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 227… find
  39. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391. find
  40. P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. 2019. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings. find
  41. Lin Mu, Xiaoyu Wang, Li Ni, Yang Li, Zhize Wu, Peiquan Jin, and Yiwen Zhang. 2025. Denselora: Dense low-rank adaptation of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 10198– 10211. find
  42. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. Lisa: Layer-wise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 37:57018–57049. find
  43. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of NAACL, pages 2080–2094.
  44. Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE. find
  45. Subhro Roy and Dan Roth. 2016. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413.
  46. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  47. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
  48. Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2025. Milora: Harnessing minor singular components for parameter-efficient llm fine-tuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tec… find
  49. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Preprint, arXiv:2312.12148. find
  50. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  51. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  52. Jinyang Zhang, Yue Fang, Hongxin Ding, Weibin Liao, Muyang Ye, Xu Chu, Junfeng Zhao, and Yasha Wang. 2025. Adept: Continual pretraining via adaptive expansion and dynamic decoupled tuning. Preprint, arXiv:2510.10071. — [arXiv:2510.10071](https://arxiv.org/abs/2510.10071) find
  53. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations (ICLR). find
  54. Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. 2025. FlyloRA: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2026-g-lora-global-local-decoupled/