Elamur Research⌕
☰
home / papers / 2026 / FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation

FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation

Paper · 2026 · Fatema Siddika, Md Anwar Hossen, Juan Pablo Munoz et al.

Summary

The paper brings Representation Fine-Tuning (ReFT) into federated learning. ReFT adapts a frozen model by editing its hidden activations, not its weights. The federated setting is hard because clients pursue different tasks, and simply averaging representation edits with FedAvg can break their semantic alignment.

The authors propose FedReFT, which builds on the low-rank variant LoReFT. Each client trains small intervention parameters (W, R, b) and sends all of them to the server. With All-But-Me (ABM) aggregation, the server gives each client a geometric median of the other clients' interventions. Each client then blends this with its own parameters, using an adaptive mixing rule inspired by Test-Time Computing. To simulate heterogeneity, the evaluation assigns different tasks from a shared benchmark family to different clients. The benchmarks cover commonsense reasoning, arithmetic reasoning and GLUE, with backbones from RoBERTa-large up to LLaMA-2 13B.

FedReFT reports state-of-the-art accuracy while training 1×–49× fewer parameters than leading federated LoRA methods. This matters for edge devices and for communication cost.

Key points

  • Method: LoReFT interventions apply Φ(h) = h + Rᵀ(Wh + b − Rh) with rank r ≪ d. Experiments showed that sharing only some of W, R and b hurt results, so FedReFT shares all three.
  • ABM aggregation: each client receives a robust geometric-median combination of every other client's updates. A TTC-style adaptive weight then balances this global signal against the client's local parameters.
  • Models and data: RoBERTa-large (GLUE), LLaMA-3.2 3B (commonsense: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA) and LLaMA-3 8B (arithmetic: GSM8K, SVAMP, AQuA, MAWPS, etc.). Other backbones include TinyLLaMA, LLaMA-2 7B/13B, Gemma-2 9B and Mistral-7B.
  • Robustness: with 5 clients all participating every round, the GLUE average was 91.45%. The paper also tests partial client participation.
  • Limitations: fixed hyperparameters and no explicit privacy mechanism; DP-SGD integration is listed as ongoing work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🏢AmazonSeattle · United States112 works · 40,565 citations🎓Iowa State UniversityAmes · United States8 works · 400 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Fatema Siddika, Md Anwar Hossen, Juan Pablo Munoz, Tanya G. Roosta, Anuj Sharma and Ali Jannesari. 2026. FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.227

@inproceedings{siddika2026fedreft,
  title = {{FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation}},
  author = {Fatema Siddika, Md Anwar Hossen, Juan Pablo Munoz, Tanya G. Roosta, Anuj Sharma and Ali Jannesari},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.227},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.227}
}

References

67 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matan Avitan, Ryan Cotterell, Yoav Goldberg, and Shauli Ravfogel. 2024. What changed? converting representational interventions to natural language. arXiv preprint arXiv:2402.11355. — [arXiv:2402.11355](https://arxiv.org/abs/2402.11355) find
  2. Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. 2024. Federated fine-tuning of large language models under heterogeneous tasks and client resources. Advances in Neural Information Processing Systems, 37:14457–14483. find
  3. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439.
  4. Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. 2017. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems (NeurIPS). find
  5. Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. 2023. Efficient federated learning for modern nlp. In Proceedings of the Annual International Conference on Mobile Computing and Networking, pages 1–16. find
  6. Jinyu Chen, Wenchao Xu, Song Guo, Junxiao Wang, Jie Zhang, and Haozhao Wang. 2022. Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers. arXiv preprint arXiv:2211.08025. — [arXiv:2211.08025](https://arxiv.org/abs/2211.08025) find
  7. Lili Chen, Lijun Su, and Jinhui Xu. 2017. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. arXiv preprint arXiv:1705.10301. — [arXiv:1705.10301](https://arxiv.org/abs/1705.10301) find
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics,…
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  10. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, and 1 others. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235.
  11. Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G Brinton. 2025. Federated sketching lora: On-device collaborative fine-tuning of large language models. arXiv preprint arXiv:2501.19389. — [arXiv:2501.19389](https://arxiv.org/abs/2501.19389) find
  12. Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. Adaptive rank selections for low-rank approximation of language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume… find
  13. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160–187. PMLR. find
  14. Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. 2024. Selective aggregation for low-rank adaptation in federated learning. arXiv preprint arXiv:2410.01463. — [arXiv:2410.01463](https://arxiv.org/abs/2410.01463) find
  15. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.
  16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML. find
  17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021a. Lora: Low-rank adaptation of large language models. arXiv preprint. ArXiv:2106.09685.
  18. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021b. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  19. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933.
  20. Rik Koncel-Kedziorski, Subhro Roy, Tao Zhang, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157. find
  21. Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowl… find
  22. Kenneth Lange. 2016. MM optimization algorithms. SIAM. find
  23. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP.
  24. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In KR. find
  25. Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024a. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159. find
  26. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024b. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36.
  27. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL. find
  28. Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2023. Relora: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Representations.
  29. Wang Ling, Dani Yogatama, Chris Dyer, and Philip Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 158–167. find
  30. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2023. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668. find
  31. Zefang Liu and Jiahua Luo. 2024. Adamole: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts. arXiv preprint arXiv:2405.00361. — [arXiv:2405.00361](https://arxiv.org/abs/2405.00361) find
  32. Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein, and 1 others. 2024. Dual-personalizing adapter for federated foundation models. Advances in Neural Information Processing Systems, 37:39409–39433. find
  33. Qikai Lu, Di Niu, Mohammadamin Samadi Khoshkho, and Baochun Li. 2024. Hyperflora: Federated learning with instantaneous personalization. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pages 824–832. find
  34. Ricardo A Maronna and Douglas Martin. 2006. Yohai. robust statistics. Wiley Series in Probability and Statistics. John Wiley and Sons, 2:3. find
  35. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR. find
  36. J. Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. 2025. Low-rank adapters meet neural architecture search for llm compression. In AAAI’25 workshop on CoL-oRAI - Connecting Low-Rank Representations in AI. find
  37. Yahao Pang, Xingyuan Wu, Xiaojin Zhang, Wei Chen, and Hai Jin. 2025. Fedeat: A robustness optimization framework for federated llms. arXiv preprint arXiv:2502.11863. — [arXiv:2502.11863](https://arxiv.org/abs/2502.11863) find
  38. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  39. Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. 2022. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142– 1154. find
  40. Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. 2024. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. arXiv preprint arXiv:2402.10462. find
  41. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  42. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
  43. Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. 2024. Mimic: Minimally modified counterfactuals in the representation space. arXiv preprint arXiv:2402.09631. — [arXiv:2402.09631](https://arxiv.org/abs/2402.09631) find
  44. Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Lav R Varshney, and Praneeth Vepakomma. 2025. Fed-sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning. arXiv preprint arXiv:2502.15436. — [arXiv:2502.15436](https://arxiv.org/abs/2502.15436) find
  45. Guangyu Sun, Matias Mendieta, Taojiannan Yang, and Chen Chen. 2022. Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning. In International Conference on Learning Representations (ICLR). find
  46. Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. 2024. Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313. — [arXiv:2403.12313](https://arxiv.org/abs/2403.12313) find
  47. Van-Tuan Tran, Quoc-Viet Pham, and 1 others. 2025. Revisiting sparse mixture of experts for resource-adaptive federated fine-tuning foundation models. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning. find
  48. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2022. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558.
  49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–35…
  50. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021. TENT: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR). find
  51. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. Adamix: Mixture-of-adaptations for parameter-efficient model tuning. arXiv preprint arXiv:2205.12410.
  52. Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. 2024. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976. — [arXiv:2409.05976](https://arxiv.org/abs/2409.05976) find
  53. Endre Weiszfeld. 1937. Sur le point pour lequel la somme des distances de n points donnés est minimum. Tohoku Mathematical Journal, First Series, 43:355– 386. find
  54. Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. 2024a. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3345–3355. find
  55. Xun Wu, Shaohan Huang, and Furu Wei. 2024b. Mixture of lora experts. arXiv preprint arXiv:2404.13628. — [arXiv:2404.13628](https://arxiv.org/abs/2404.13628) find
  56. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2024c. Reft: Representation fine-tuning for language models. arXiv preprint. ArXiv:2404.03592. find
  57. Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, and Lei Zhu. Federated residual low-rank adaptation of large language models. In The Thirteenth International Conference on Learning Representations. find
  58. Yifan Yang, Kai Zhen, Ershad Banijamal, Athanasios Mouchtaris, and Zheng Zhang. 2024. Adazeta: Adaptive zeroth-order tensor-train adaption for memory-efficient large language models fine-tuning. arXiv preprint arXiv:2406.18060. find
  59. Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. 2023. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283. — [arXiv:2310.13283](https://arxiv.org/abs/2310.13283) find
  60. Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. 2018. Byzantine-robust distributed learning: Towards optimal statistical rates. In International conference on machine learning, pages 5650–5659. Pmlr. find
  61. Ben Zaken, Yoav Goldberg, and Amir Globerson. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In ACL Findings.
  62. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  63. Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. 2024. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915–6919. IEE… find
  64. Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. 2023. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics (ACL), pages 9963–9977. find
  65. Changhai Zhou, Shijie Han, Shiyang Zhang, Shichao Weng, Zekai Liu, and Cheng Jin. 2024. Rankadaptor: Hierarchical dynamic low-rank adaptation for structural pruned llms. arXiv preprint arXiv:2406.15734. — [arXiv:2406.15734](https://arxiv.org/abs/2406.15734) find
  66. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find
  67. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siddika-2026-fedreft-federated-representation-fine-tuning/