Elamur Research⌕
☰
home / papers / 2026 / Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

Summary

Federated learning (FL) of multilingual LLMs is difficult when clients hold text in different languages. These non-IID splits cause client drift and tend to disadvantage languages with fewer resources. The authors study how the language mix inside each client's local data affects global model quality, cross-lingual fairness and training cost. They extend the FederatedScope-LLM framework with multilingual data pipelines, language-aware prompt handling, and separate client and server data splits. They also add LDES-FL, an early-stopping scheme in which each client suspends and later resumes local training based on its own validation loss. Using LoRA adapters merged with equally weighted FedAvg, they instruction-tune salamandra-2b-instruct across eight clients, ranging from strictly monolingual clients to clients with growing shares of other languages. Purely local monolingual tuning remains best for specializing in one language, while FL is the better route to a single balanced multilingual model. More multilingual clients produce a stronger and fairer global model that comes closer to centralized multilingual fine-tuning, with the largest benefit for lower-resource languages, though more optimization steps are needed. The study identifies client language composition as a core design choice in privacy-preserving multilingual training.

Key points

  • Released code: the FederatedScope-LLM extension is available as the FedEloquence repository (github.com/Telefonica-Scientific-Research/FedEloquence).
  • LDES-FL: a client-level early-stopping mechanism. Each client pauses or restarts local training according to its local validation loss, which saves compute and makes training dynamics easier to analyse.
  • Setup:
    • Model: salamandra-2b-instruct, tuned with LoRA (only adapter matrices are communicated and averaged).
    • Clients: eight, each with a dominant language plus equal smaller portions of the other languages.
    • Aggregation: FedAvg with identical client weights, because all local training sets are the same size.
  • Main trend: greater within-client multilinguality reduces drift, raises average multilingual performance, narrows the gap to centralized training, and helps lower-resource languages most, at the cost of more optimization steps.
  • Missing details: exact scores, the language list and the evaluation datasets were in the portion of the PDF omitted from the extracted text. Consult the full paper for concrete numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4eyoaxvbuw3y

@inproceedings{sant2026optimizing,
  title = {{Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition}},
  author = {Sant, Aleix and Luque, Jordi and Escolano, Carlos},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4eyoaxvbuw3y},
  url = {https://doi.org/10.63317/4eyoaxvbuw3y}
}

References

37 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yujun Cheng, Weiting Zhang, Zhewei Zhang, Chuan Zhang, Shengjin Wang, and Shiwen Mao. 2025. Toward Federated Large Language Models: Motivations, Methods, and Future Directions. IEEE Communications Surveys Tutorials, 27(4):2733–2764. find
  2. Ning Ding, Bing Qin, and Ting Liu. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. In Findings of the Association for Computational Linguistics (ACL Findings). find
  3. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, Mario Mina, Iñigo Pikabea, Adrián Rubio, Alexander Shvets, Anna Sallés, Iñaki Lacunza, Jorge Palomar, Júlia Falcão, Lucía Tormo, et al. 20… — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) find
  4. Demi Guo, Alexander M Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  5. Zhihan Guo, Yifei Zhang, Zhuo Zhang, Zenglin Xu, and Irwin King. 2024. FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 1519–152… find
  6. Gururise. 2023. Cleaned Alpaca Dataset. Accessed: 2025-09-06. find
  7. Agrin Hilmkil, Sebastian Callh, Matteo Barbieri, Leon René Sütfeld, Edvin Listo Zec, and Olof Mogren. 2021. Scaling Federated Learning for Fine-Tuning of Large Language Models. In Natural Language Processing and Information Systems - 26th International Conference on Applications of Natural Language… find
  8. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Brianna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). find
  9. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find
  10. Jiahui Hu, Dan Wang, Zhibo Wang, Xiaoyi Pang, Huiyu Xu, Ju Ren, and Kui Ren. 2025. Federated Large Language Model: Solutions, Challenges and Future Directions. IEEE Wireless Communications, 32(4):82–89. find
  11. Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B.… — [arXiv:1912.04977](https://arxiv.org/abs/1912.04977) find
  12. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. 2019. SCAFFOLD: stochastic controlled averaging for on-device federated learning. CoRR, abs/1910.06378. — [arXiv:1910.06378](https://arxiv.org/abs/1910.06378) find
  13. Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowl… find
  14. Royson Lee, Minyoung Kim, Fady Rezk, Rui Li, Stylianos I. Venieris, and Timothy M. Hospedales. 2025. FedP2 EFT: Federated Learning to Personalize Parameter Efficient Fine-Tuning for Multilingual LLMs. CoRR, abs/2502.04387. — [arXiv:2502.04387](https://arxiv.org/abs/2502.04387) find
  15. Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. In Proceedings of the Third Conference on Machine Learning and Systems, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org. find
  16. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  17. Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, Matt White, and Meikang Qiu. 2025. Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning. ACM Trans. Manag. Inf. Syst., 16(2):1– 24. find
  18. Zili Lu, Heng Pan, Yueyue Dai, Xueming Si, and Yan Zhang. 2024. Federated Learning With Non-IID Data: A Survey. IEEE Internet of Things Journal, 11(11):19188–19209. find
  19. Andre Manoel, Mirian del Carmen Hipolito Garcia, Tal Baumel, Shize Su, Jialei Chen, Robert Sim, Dan Miller, Danny Karmon, and Dimitrios Dimitriadis. 2023. Federated Multilingual Models for Medical Transcript Analysis. In Conference on Health, Inference, and Learning, CHIL 2023, Broad Institute of MI… find
  20. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Le… find
  21. H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. 2016. Federated Learning of Deep Networks using Model Averaging. CoRR, abs/1602.05629. — [arXiv:1602.05629](https://arxiv.org/abs/1602.05629) find
  22. Matías Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. 2022. Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 8387–83… find
  23. Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H. Brendan McMahan. 2020. Adaptive Federated Optimization. CoRR, abs/2003.00295. — [arXiv:2003.00295](https://arxiv.org/abs/2003.00295) find
  24. Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. 2018. On the Convergence of Federated Optimization in Heterogeneous Networks. CoRR, abs/1812.06127. — [arXiv:1812.06127](https://arxiv.org/abs/1812.06127) find
  25. Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. 2023. Towards Personalized Federated Learning. IEEE Trans. Neural Networks Learn. Syst., 34(12):9587–9603. find
  26. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca: A Strong, Replicable Instruction-Following Model. Accessed: 2025-09-06. find
  27. Bibek Upadhayay and Vahid Behzadan. 2023. TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes. CoRR, abs/2311.10797. find
  28. Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. 2020. Optimizing Federated Learning on Non-IID Data with Reinforcement Learning. In 39th IEEE Conference on Computer Communications, INFOCOM 2020, Toronto, ON, Canada, July 6-9, 2020, pages 1698–1707. IEEE. find
  29. Haoyu Wang, Handong Zhao, Yaqing Wang, Tong Yu, Jiuxiang Gu, and Jing Gao. 2022. FedKC: Federated Knowledge Composition for Multilingual Natural Language Understanding. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 1839–1850, New York, NY, USA. Association for Computing Machinery. find
  30. Orion Weller, Marc Marone, Vladimir Braverman, Dawn J. Lawrie, and Benjamin Van Durme. 2022. Pretrained Models for Multilingual Federated Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL… find
  31. Mang Ye, Xiuwen Fang, Bo Du, Pong C. Yuen, and Dacheng Tao. 2024a. Heterogeneous Federated Learning: State-of-the-art and Research Challenges. ACM Comput. Surv., 56(3):79:1–79:44. find
  32. Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. 2024b. FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 20… find
  33. Han Zhang et al. 2023. Multilingual Parameter-Efficient Fine-tuning of Large Language Models. arXiv preprint arXiv:2305.12345. — [arXiv:2305.12345](https://arxiv.org/abs/2305.12345) find
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  35. Wanru Zhao, Yihong Chen, Royson Lee, Xinchi Qiu, Yan Gao, Hongxiang Fan, and Nicholas D. Lane. 2025. Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages. CoRR, abs/2507.03003. — [arXiv:2507.03003](https://arxiv.org/abs/2507.03003) find
  36. Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018. Federated Learning with Non-IID Data. CoRR, abs/1806.00582. — [arXiv:1806.00582](https://arxiv.org/abs/1806.00582) find
  37. Jiaying Zheng, Hainan Zhang, Lingxiang Wang, Wangjie Qiu, Hong-Wei Zheng, and Zhi Ming Zheng. 2024. Safely Learning with Private Data: A Federated Learning Framework for Large Language Model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sant-2026-optimizing-multilingual-llms-federated-learning/