Elamur Research⌕
☰
home / papers / 2026 / Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing

Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing

Paper · 2026 · Haoran Wang, Xiong Wang, Yuqing Li et al.

Summary

Federated LoRA lets several clients jointly adapt an LLM without sharing raw data. Standard systems average the two low-rank factors B and A separately under FedAvg. This introduces a systematic aggregation error, because the product of averaged factors is not the average of the per-client products. Edge clients also face heavy communication and compute costs. The authors propose iFLoRA to address both problems. The server rebuilds each client's full update as BA, averages these updates, and factorizes the result with SVD. It then keeps only the important singular values, which sets the LoRA rank for the next round. To hide the extra SVD cost, server aggregation runs per matrix in parallel with clients' backward passes and uploads. A matrix-wise freezing scheme detects weight matrices that have stabilized and pauses them for adaptively chosen periods. Experiments used BERT-large, RoBERTa and Llama-3.2-3B on 20NEWS, AGNEWS, SQuAD2.0 and SAMSum. iFLoRA reached target accuracy 2.17–8.48× faster than prior methods and sharply cut network traffic. This makes collaborative, privacy-preserving LLM tuning more practical on resource-limited devices.

Key points

  • Baselines and traffic: iFLoRA was compared with FedPETuning, FFA-LoRA and FedEx-LoRA. It reduced network traffic by 74.47–97.96%, 90.19–97.81% and 94.09–99.36% respectively, and lowered client memory use by 9.42–25.47%.
  • Ablation on BERT-large / 20NEWS:
    • SVD-based aggregation with importance-aware rank gave a 2.18× speedup.
    • Freezing alone gave 2.12×.
    • The full system (aggregation, pipelining and freezing) reached 2.37×.
  • SVD overhead: SVD took about 68.04 ms per aggregated matrix for Llama-3.2-3B on RTX 4090 GPUs. Storing low-rank factors instead of full updates cut server memory for freezing from 75 GB to 1.03 GB.
  • Motivating experiment (Llama-3.2-3B on SQuAD2.0):
    • Layers near the input converged earlier than those near the output.
    • Freezing the first 12 layers lowered F1 only from 92.89% to 91.16%.
  • Limitations: Differential privacy was not evaluated. The matrix multiplication and SVD steps are incompatible with current secure aggregation protocols. Code is released on GitHub (whr819987540/iflora).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations🎓Wuhan UniversityWuhan · China35 works · 1,127 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Haoran Wang, Xiong Wang, Yuqing Li, Jing Chen, Junyi Zhang, Nan Yan, Kun He and Wei Wang. 2026. Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.284

@inproceedings{wang2026federated,
  title = {{Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing}},
  author = {Haoran Wang, Xiong Wang, Yuqing Li, Jing Chen, Junyi Zhang, Nan Yan, Kun He and Wei Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.284},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.284}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. 2021. Communication-efficient federated learning with adaptive parameter freezing. In Proceedings of the 41st IEEE International Conference on Distributed Computing Systems (ICDCS), pages 1–11. find
  3. Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. 2024. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12903–12913. find
  4. Rohan Deb, Kiran Koshy Thekumparampil, Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, and Branislav Kveton. 2025. Fishersft: Data-efficient supervised fine-tuning of language models using information gain. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 1… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  6. Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2665–267… find
  7. Carl Eckart and Gale Young. 1936. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218. find
  8. Zhidong Gao, Zhenxiao Zhang, Yuanxiong Guo, and Yanmin Gong. 2025. Federated adaptive fine-tuning of large language models with heterogeneous quantization and lora. In Proceedings of IEEE INFOCOM 2025 - IEEE Conference on Computer Communications, pages 1–10. find
  9. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79.
  10. Per Christian Hansen. 1987. The truncated svd as a method for regularization. BIT Numerical Mathematics, 27(4):534–553. find
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR).
  12. Ian T Jolliffe. 2002. Principal Component Analysis for Special Types of Data. Springer New York, New York, NY. find
  13. Fan Lai, Yinwei Dai, Sanjay Sri Vallabh Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. 2022. Fedscale: Benchmarking model and system performance of federated learning at scale. In Proceedings of the 39th International Conference on Machine Learning (ICML), page… find
  14. Fan Lai, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. 2021. Oort: Efficient federated learning via guided participant selection. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 19–35. find
  15. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the 12th International Conference on Machine Learning (ICML), pages 331– 339. find
  16. Chenning Li, Xiao Zeng, Mi Zhang, and Zhichao Cao. 2022. Pyramidfl: a fine-grained client selection framework for efficient federated learning. In Proceedings of ACM MobiCom ’22: the 28th Annual International Conference on Mobile Computing and Networking, pages 158–171. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  18. Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. 2021. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXiv:1907.11692.
  19. Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. 2022. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In Proceedings of IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, pages 1739–1748. find
  20. M-Lab. 2022. The m-lab mobiperf data set. find
  21. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. find
  22. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1273–1282. find
  23. Sachin Mehta, Mohammad Sekhavat, Qingqing Cao, Max Horton, Yanzi Jin, Frank Sun, Iman Mirzadeh, Mahyar Najibikohnehshahri, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. Openelm: An efficient language model family with open training and inference framework. In Proceedings of the 2nd… find
  24. NVIDIA. 2025. Nvidia jetson modules. find
  25. Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, and Cheng Wang. 2024. Fedpft: Federated proxy fine-tuning of foundation models. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI), pages 4806–4814. find
  26. Wenqi Qiu, Yipeng Zhou, Jinzhi Wang, Quan Z. Sheng, and Laizhong Cui. 2025. Flm-topk: Expediting federated large language model tuning by sparsifying intervalized gradients. In Proceedings of IEEE IN-FOCOM 2025 - IEEE Conference on Computer Communications, pages 1–10. find
  27. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 6076–6085. find
  28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 784–789.
  29. Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. 2024. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pa… find
  30. Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang. 2022. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI), pages 2218–2224. find
  31. Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. 2025. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1316–1336. learning. In Proceedings… find
  32. Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. 2022. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 21111–21132. find
  33. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  34. Madeleine Udell and Alex Townsend. 2019. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144– 160. find
  35. Yiding Wang, Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowdhury. 2023. Egeria: Efficient DNN training with knowledge-guided layer freezing. In Proceedings of the 18th European Conference on Computer Systems (EuroSys), pages 851–866. find
  36. Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. 2024. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 22513– 22533. find
  37. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023a. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations. find
  38. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  39. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 28, pages 649–657.
  40. Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. 2023b. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics (ACL), pages 9963–9977. find
  41. Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. 2024. Improving lora in privacy-preserving federated find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-federated-lora-fine-tuning-pipelined/