Elamur Research⌕
☰
home / papers / 2025 / FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models

FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models

Paper · 2025 · Tao Fan, Guoqiang Ma, Yan Kang et al.

Summary

Federated work on LLMs has mostly followed two separate tracks. In one, clients jointly fine-tune identical LLMs. In the other, a server-side LLM teaches small language models (SLMs) on clients. Neither improves the server LLM and the client SLMs together. FedMKT, a parameter-efficient framework, makes knowledge flow both ways. In every round:

  • clients fine-tune their SLMs on private data;
  • clients send output logits computed on a shared public dataset;
  • the server distils this client knowledge into its LLM and returns its own logits for the SLMs to learn from.

Two ingredients make this work. First, tokenizers differ across models (e.g., Bloom vs. LLaMa), so tokens are aligned with minimum edit distance (MinED). Second, a selective step (DualMinCE) keeps another model's predictions only when that model's loss on a sample is lower than the local one.

The authors test three configurations: Heterogeneous, Homogeneous and One-to-One, using public models such as LLaMa2-7B, LLaMa2-1.3B and OPT-1.3B. The client SLMs improve over standalone and FedAvg training. The server LLM comes close to centralized fine-tuning on pooled data without collecting raw client data, which is relevant for privacy-constrained domain adaptation.

Key points

  • Server model is LLaMa2-7B. Clients use LLaMa2-1.3B or OPT-1.3B, with four clients in the Homogeneous setups and one in One-to-One. Code is released in the FATE-LLM project (FATE).
  • Evaluation is zero-shot with lm-evaluation-harness. QA tasks (CQA, ARC-C, ARC-E) use Accuracy, and instruction-following tasks use Rouge-L.
  • On CQA in the Homogeneous setting, the server LLM rises from 39.5 (Zero-Shot) to 68.8/71.3 with FedMKT. Centralized training reaches 69.5.
  • Client SLMs on CQA gain roughly 4% (LLaMa2-1.3B: 58.6 vs. 56.4) and 5% (OPT-1.3B: 61.5 vs. 58.6) relative to FedAvg.
  • The authors list three limitations:
    • Sharing logits has no formal privacy guarantee.
    • Spammer clients are not handled.
    • Compute limits kept them from testing larger models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen and Qiang Yang. 2025. FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fan2025fedmkt,
  title = {{FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models}},
  author = {Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen and Qiang Yang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.17/}
}

References

44 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. 2022. Aut-ofednlp: An efficient fednlp framework. arXiv preprint arXiv:2205.10162. — [arXiv:2205.10162](https://arxiv.org/abs/2205.10162) find
  2. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. Dialogsum: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06762.
  3. Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. 2021. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems, 36(6):87–98. find
  4. Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. 2022. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703. — [arXiv:2204.12703](https://arxiv.org/abs/2204.12703) find
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  7. Tao Fan, Weijing Chen, Guoqiang Ma, Yan Kang, Lixin Fan, and Qiang Yang. 2024a. Secureboost+: Large scale and high-performance vertical federated gradient boosting decision tree. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 237–249. Springer. find
  8. Tao Fan, Yan Kang, Weijing Chen, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen, and Qiang Yang. 2024b. Pdss: A privacy-preserving framework for step-by-step distillation of large language models. arXiv preprint arXiv:2406.12403. — [arXiv:2406.12403](https://arxiv.org/abs/2406.12403) find
  9. Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049. — [arXiv:2310.10049](https://arxiv.org/abs/2310.10049) find
  10. Tao Fan, Yan Kang, Guoqiang Ma, Lixin Fan, Kai Chen, and Qiang Yang. 2024c. Fedcollm: A parameter-efficient federated co-tuning framework for large and small language models. arXiv preprint arXiv:2411.11707. find
  11. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pages 10421–10430. PMLR.
  12. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  13. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations. find
  14. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.
  15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. find
  17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  18. Jaehee Jang, Heoneok Ha, Dahuin Jung, and Sungroh Yoon. 2022. Fedclassavg: Local representation learning for personalized federated learning on heterogeneous neural networks. In Proceedings of the 51st International Conference on Parallel Processing, pages 1–10. find
  19. Yan Kang, Tao Fan, Hanlin Gu, Lixin Fan, and Qiang Yang. 2023. Grounding foundation models through federated transfer learning: A general framework. arXiv preprint arXiv:2311.17431. — [arXiv:2311.17431](https://arxiv.org/abs/2311.17431) find
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  21. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — Datasets: A Community Library for Natural Language Processing
  22. Daliang Li and Junpu Wang. 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581. — [arXiv:1910.03581](https://arxiv.org/abs/1910.03581) find
  23. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  24. Chang Liu, Yuwen Yang, Xun Cai, Yue Ding, and Hongtao Lu. 2022. Completely heterogeneous federated learning. arXiv preprint arXiv:2210.15865. — [arXiv:2210.15865](https://arxiv.org/abs/2210.15865) find
  25. Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. 2021. Fate: An industrial grade platform for collaborative learning with data protection. J. Mach. Learn. Res., 22(226):1–6. find
  26. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/ peft. ↗
  27. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR. find
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  29. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  30. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937.
  31. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  33. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. find
  34. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537.
  35. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2.… find
  36. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694. find
  37. Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. 2019. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1–207. 5. On the server side, knowledge is selected from the SLMs to the LLM according to Algorithm 2. find
  38. Ruihong Yang, Junchao Tian, and Yu Zhang. 2021. Regularized mutual learning for personalized federated learning. In Asian Conference on Machine Learning, pages 1521–1536. PMLR. 6. On the server side, knowledge is transferred from the SLMs to the LLM based on the selected knowledge. find
  39. Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. 2023. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283. 7. Once the knowledge transfer from SLMs to LLM is completed on the server, the server then computes a knowledge set S 0 = {l 0 i , p… — [arXiv:2310.13283](https://arxiv.org/abs/2310.13283) find
  40. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. \#\# 8. The server disseminates S 0 to all the clients. 9. On the…
  41. Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320–4328. 10. On the client side, knowledge is selected from the LLM to each client SLM according to Algorithm 2. find
  42. Zhuo Zhang, Yuanhang Yang, Yong Dai, Lizhen Qu, and Zenglin Xu. 2022b. When federated learning meets pre-trained language models’ parameter-efficient tuning methods. arXiv preprint arXiv:2212.10025. — [arXiv:2212.10025](https://arxiv.org/abs/2212.10025) find
  43. Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. 2022. Reduce communication costs and preserve privacy: Prompt tuning method in federated learning. arXiv preprint arXiv:2208.12268. — [arXiv:2208.12268](https://arxiv.org/abs/2208.12268) find
  44. Yin Zhu, Xiao Wang, Erheng Zhong, Nathan Liu, He Li, and Qiang Yang. 2012. Discovering spammers in social networks. In proceedings of the AAAI conference on artificial intelligence, volume 26, pages 171–177. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fan-2025-fedmkt-federated-mutual-knowledge-transfer/