Elamur Research⌕
☰
home / papers / 2026 / Safe-FedLLM: Delving into the Safety of Federated Large Language Models

Safe-FedLLM: Delving into the Safety of Federated Large Language Models

Paper · 2026 · Mingxiang Tao, Yu Tian, Wenxuan Tu et al.

Summary

Federated fine-tuning of LLMs (FedLLM) lets parties train jointly without pooling raw data. Little is known about how to defend such systems against malicious clients when the server sees only LoRA updates. The authors built a testbed on OpenFedLLM and FedLLM-Attack using Llama3.1-8B and Qwen2.5-7B. Benign clients train on LMSYS-Chat or WildChat, and harmful clients train on BeaverTails or MaliciousGen. Two findings emerged. Even a modest share of attackers sharply erodes safety. Benign and malicious LoRA updates also form clearly separable clusters. Building on this, they propose Safe-FedLLM. It uses an offline-trained, frozen LoRA-Probe classifier that treats each client's ∆LoRA as a behavioral feature vector and down-weights or filters suspicious contributions before aggregation. The defense works at three granularities: Step-Level, Client-Level and Shadow-Level. Across backbones, data sources and attacker ratios, safety improves substantially, utility stays competitive and training slows only slightly. This matters because it shows that model updates alone can serve as an internal safety signal for privacy-preserving LLM training.

Key points

  • Evaluation setup: Safety is measured on AdvBench with three metrics: Rule, MD-Judge and a reward model (RM). Utility is measured with MT-1, the first-turn MT-Bench score judged by GPT-5-mini. Malicious client ratios range from 20% to 50%.
  • Vulnerability of plain FedAvg: With MaliciousGen and LMSYS-Chat on Llama3.1-8B, the Rule safety score falls from 90.77 with no attackers to 60.77 at a 20% malicious ratio.
  • Gains with a 30% malicious ratio (BeaverTails & LMSYS-Chat): Rule rises from 51.73 under FedAvg to 91.92 with Shadow-Level. Shadow-Level gives the strongest and most round-stable protection, keeping Rule at about 92–94 even with 50% attackers.
  • Overhead:
    • Training time increases by about 3.2%.
    • Step-Level and Client-Level keep 3.41M LoRA parameters.
    • Shadow-Level doubles this to 6.82M and needs 41.24GB of GPU memory instead of 20.62GB, but its extra branch runs in parallel.
  • Limitations:
    • Clients must use the same LoRA initialization seed as the probe.
    • The probe transfers poorly across backbones.
    • Probe precision for Step-Level and Client-Level declines in later rounds as the global model evolves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Hainan UniversityHaikou · China3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Networks and Communications · 64 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Mingxiang Tao, Yu Tian, Wenxuan Tu, Yue Yang, Xue Yang and Xiangyan Tang. 2026. Safe-FedLLM: Delving into the Safety of Federated Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1120

@inproceedings{tao2026safe,
  title = {{Safe-FedLLM: Delving into the Safety of Federated Large Language Models}},
  author = {Mingxiang Tao, Yu Tian, Wenxuan Tu, Yue Yang, Xue Yang and Xiangyan Tang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1120},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1120}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020. How to backdoor federated learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 2938–2948. find
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  3. Moran Baruch, Gilad Baruch, and Yoav Goldberg. 2019. A little is enough: Circumventing defenses for distributed learning. In Advances in Neural Information Processing Systems. find
  4. Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. 2019. Analyzing federated learning through an adversarial lens. In Proceedings of the International Conference on Machine Learning, pages 634–646. find
  5. Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. 2017. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems. find
  6. Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. 2021. FLTrust: Byzantine-robust federated learning via trust bootstrapping. In Proceedings of the Network and Distributed System Security Symposium. find
  7. Xiaoyu Cao and Neil Zhenqiang Gong. 2022. MPAF: Model poisoning attacks to federated learning based on fake clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 3396–3404. find
  8. Jiawei Chen, Xiao Yang, Zhengwei Fang, Yu Tian, Yinpeng Dong, Zhaoxia Yin, and Hang Su. 2025. Auto-breach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization via multi-LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6792–6813. and Wi… find
  9. Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi… find
  10. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations.
  11. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Open… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  12. El-Mahdi El-Mhamdi, Rachid Guerraoui, and Sébastien Rouault. 2018. The hidden vulnerability of distributed learning in byzantium. In Proceedings of the International Conference on Machine Learning, pages 3518–3527. find
  13. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
  14. Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. 2020. Local model poisoning attacks to byzantine-robust federated learning. In Proceedings of the USENIX Security Symposium, pages 1605–1622. find
  15. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. SALAD-bench: A hierarchical and comprehensive safety benchmark for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3923–3954.
  16. Shuhao Fu, Chulin Xie, Bo Li, and Qifeng Chen. 2019. Attack-resistant federated learning with residual-based reweighting. arXiv preprint arXiv:1912.11464. — [arXiv:1912.11464](https://arxiv.org/abs/1912.11464) find
  17. Clement Fung, Chris J.M. Yoon, and Ivan Beschast-nikh. 2020. The limitations of federated learning in sybil settings. In Proceedings of the International Symposium on Research in Attacks, Intrusions and Defenses, pages 301–316. find
  18. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  19. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. BeaverTails: Towards improved safety alignment of llm via a human-preference dataset. In Advances in Neural Information Processing Systems.
  20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Jingxin Liu, Jieren Cheng, Renda Han, Wenxuan Tu, Jiaxin Wang, and Xin Peng. 2025a. Federated graph-level clustering network. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18870– 18878. find
  22. Jingxin Liu, Renda Han, Wenxuan Tu, Haotian Wang, Junlong Wu, and Jieren Cheng. 2025b. Federated node-level clustering network with cross-subgraph link mending. In Proceedings of the International Conference on Machine Learning, pages 38540– 38556. find
  23. Jingxin Liu, Wenxuan Tu, Renda Han, Junlong Wu, Haotian Wang, Guohui Liu, Xiangyan Tang, and Yue Yang. 2026a. Personalized federated graph-level clustering network. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23721–23729. find
  24. Jingxin Liu, Wenxuan Tu, Haotian Wang, Renda Han, Haoyi Li, Junlong Wu, and Xiangyan Tang. 2026b. Causally-aware attribute completion for incomplete federated graph clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23730–23738. find
  25. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The FLAN collection: Designing data and methods for effective instruction tuning. In Proceedings of the International Conference on Machine Learning, pages 2…
  26. H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 1273–1282. find
  27. Meta AI. 2024. Llama 3.1 model card. find
  28. Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, Farinaz Koushanfar, Ahmad-Reza Sadeghi, and Thomas Schneider. 2022. FLAME: Taming backdoors in federated learning. In Proceedings of… find
  29. OpenAI. 2025. GPT-5 system card. find
  30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  31. Krishna Pillutla, Sham Kakade, and Zaid Harchaoui. 2022. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142– 1154. find
  32. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  33. Virat Shejwalkar and Amir Houmansadr. 2021. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In Proceedings of the Network and Distributed System Security Symposium. find
  34. Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. 2025. FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1316–1336. find
  35. Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of LLM-based agents. arXiv preprint arXiv:2311.11855. — [arXiv:2311.11855](https://arxiv.org/abs/2311.11855) find
  36. Lingren Wang, Wenxuan Tu, Jieren Cheng, Jianan Wang, Xiangyan Tang, and Chenchen Wang. 2025a. Discovering maximum frequency consensus: Lightweight federated learning for medical image segmentation. In Proceedings of the ACM International Conference on Multimedia, pages 1900–1909. find
  37. Lingren Wang, Wenxuan Tu, Jiaxin Wang, Xiong Wang, Jieren Cheng, and Jingxin Liu. 2025b. FedIGL: Federated invariant graph learning for non-iid graphs. In Advances in Neural Information Processing Systems. find
  38. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  39. Jiahao Xu, Zikai Zhang, and Rui Hu. 2025. Achieving byzantine-resilient federated learning via layer-adaptive sparsified model aggregation. In Proceedings of the Winter Conference on Applications of Computer Vision. find
  40. Rui Ye, Jingyi Chai, Xiangrui Liu, Yaodong Yang, Yanfeng Wang, and Siheng Chen. 2025. Emerging safety attack and defense in federated instruction tuning of large language models. In International Conference on Learning Representations. find
  41. Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. 2024. OpenFedLLM: Training large language models on decentralized private data via federated learning. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Min… find
  42. Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. 2018. Byzantine-robust distributed learning: Towards optimal statistical rates. In Proceedings of the International Conference on Machine Learning, pages 5650–5659. find
  43. Moxuan Zeng, Wenxuan Tu, Yuanyi Chen, Yiying Wang, Miao Yu, Xiangyan Tang, and Jieren Cheng. 2026. FedPKDA: Personalized federated learning with privacy-preserving knowledge dynamic alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 28113–28121. find
  44. Xinyi Zeng, Yuying Shang, Jiawei Chen, Jingyuan Zhang, and Yu Tian. 2025. Root defense strategies: Ensuring safety of LLM at the decoding level. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1974–1988. find
  45. Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. 2024. Towards building the federatedgpt: Federated instruction tuning. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6915–6919. find
  46. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. WildChat: 1m ChatGPT interaction logs in the wild. In International Conference on Learning Representations.
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E Gonzalez, Ion Stoica, and Hao Zhang. 2024. LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset. In International Conference on Learning Representatio…
  48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems.
  49. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tao-2026-safe-fedllm-delving-into-safety/