Elamur Research⌕
☰
home / papers / 2024 / Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

Paper · 2024 · Enneng Yang, Li Shen, Guibing Guo et al.

Summary

Model merging combines the weights of several already-trained expert networks into one model, usually needing neither the source training data nor costly retraining. The authors argue that earlier overviews were narrow. Benchmark toolkits such as MergeKit and FusionBench cover only a handful of techniques, so they write a broad survey rather than release a new dataset or algorithm. They sort the literature by when an intervention happens: before merging, which prepares models so they combine well, or during merging, which is the fusion procedure itself. They also review theoretical and empirical analyses. They then map uses across LLMs, MLLMs, image generation models and more than ten machine-learning subfields, including continual, multi-task, few-shot, federated and adversarial learning. The survey ends with seven open problems. Merged models still trail individual experts noticeably, merging consumes a lot of memory, and theory and trust guarantees are thin. For practitioners adapting open checkpoints, including for Ukrainian NLP, it offers a structured map of low-cost ways to combine fine-tuned models, backed by a maintained paper list.

Key points

  • The taxonomy has two stages. The pre-merging stage covers merger-friendly fine-tuning, architecture transformation, and weight re-basin/alignment. The during-merging stage covers basic, weighted, subspace, optimization-based, routing-based and post-calibration methods.
  • The survey reports a large gap between merged models and experts. On eight vision tasks, separately fine-tuned ViT-B/32 models score 90.3, while plain weight averaging reaches 66.5. For Llama-3.1, the best merging method scores 84.8 against the experts' 100.
  • The authors flag efficiency as a concern: merging 72 fine-tuned ViT-B/32 models can require over 200GB of memory.
  • The LLM use cases include value alignment, detoxification, knowledge unlearning, and faster training, inference and reasoning. MLLM use cases include multimodal fusion and cross-modal transfer, and visual generative models use merging for style mixing.
  • The companion paper list is hosted on GitHub as "Awesome-Model-Merging-Methods-Theories-Applications" (EnnengYang).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China72 works · 9,502 citations🎓Northeastern UniversityShenyang · China53 works · 2,205 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang and Dacheng Tao. 2024. Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities. arXiv:2408.07666.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.07666 · arXiv 2408.07666

@misc{yang2024model,
  title = {{Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities}},
  author = {Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang and Dacheng Tao},
  year = {2024},
  doi = {10.48550/arxiv.2408.07666},
  eprint = {2408.07666},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.07666}
}

References

292 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. 2024. Strong Copyright Protection for Language Models via Adaptive Model Fusion. ICML (2024). find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
  3. Linara Adilova, Asja Fischer, and Martin Jaggi. 2024. Layerwise linear mode connectivity. ICLR (2024). find
  4. Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, and Barlas Oguz. 2024. Jointly training large autoregressive multimodal models. ICLR (2024). find
  5. Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git Re-Basin: Merging Models modulo Permutation Symmetries. In ICLR. find
  6. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2025. Evolutionary optimization of model merging recipes. Nature Machine Intelligence 7, 2 (2025), 195–204. find
  7. Anton Alexandrov, Veselin Raychev, Mark Niklas Müller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. Mitigating Catastrophic Forgetting in Language Transfer via Model Merging. arXiv:2407.08699 find
  8. Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. 2022. Ensemble of averages: Improving model selection and boosting performance in domain generalization. NeurIPS 35 (2022), 8265–8277. find
  9. Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. 2024. Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? arXiv preprint arXiv:2402.15414 (2024). — [arXiv:2402.15414](https://arxiv.org/abs/2402.15414) find
  10. Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022. GAN Cocktail: mixing GANs without dataset access. In ECCV. Springer, 205–221. find
  11. Robert Belanec, Simon Ostermann, Ivan Srba, and Maria Bielikova. 2024. Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer. arXiv preprint arXiv:2408.01119 (2024). — [arXiv:2408.01119](https://arxiv.org/abs/2408.01119) find
  12. Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. 2021. Loss surface simplexes for mode connecting volumes and fast ensembling. In ICML. PMLR, 769–779. find
  13. Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. arXiv preprint arXiv:2402.11746 (2024).
  14. Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Diffusion Soup: Model Merging for Text-to-Image Diffusion Models. arXiv preprint arXiv:2406.08431 (2024). — [arXiv:2406.08431](https://arxiv.org/abs/2406.08431) find
  15. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In WWW. 491–500. find
  16. Ruisi Cai, Zhenyu Zhang, and Zhangyang Wang. 2023. Robust weight signatures: gaining robustness as easy as patching weights?. In ICML. PMLR, 3495–3506. find
  17. Sheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian, and Zechun Liu. 2025. ParamΔ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost. CoRR abs/2504.21023 (2025). — [arXiv:2504.21023](https://arxiv.org/abs/2504.21023) find
  18. Rich Caruana. 1997. Multitask learning. Machine learning 28 (1997), 41–75. find
  19. Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. 2021. Swad: Domain generalization by seeking flat minima. NeurIPS 34 (2021), 22405–22418. find
  20. Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, et al. 2024. Model Composition for Multimodal Large Language Models. ACL (2024). find
  21. Guangyao Chen, Peixi Peng, Yangru Huang, Mengyue Geng, and Yonghong Tian. 2024. Adaptive Discovering and Merging for Incremental Novel Class Discovery. In AAAI, Vol. 38. 11276–11284. find
  22. Weiyu Chen and James Kwok. 2025. Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging. In ICML. find
  23. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187 (2024).
  24. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML. PMLR, 794–803. find
  25. Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. 2024. DAM: Dynamic Adapter Merging for Continual Video QA Learning. arXiv preprint arXiv:2403.08755 (2024). find
  26. Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. 2025. Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors. In ICML. find
  27. Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Ghotkar. 2023. Task Arithmetic with LoRA for Continual Learning. arXiv preprint arXiv:2311.02428 (2023). — [arXiv:2311.02428](https://arxiv.org/abs/2311.02428) find
  28. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113.
  29. Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. 2023. AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models. In EACL. 2009–2018. find
  30. Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2023. Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization. arXiv preprint arXiv:2311.09344 (2023). find
  31. Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. 2024. Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging. arXiv preprint arXiv:2404.05188 (2024). find
  32. Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. 2024. 𝐶 2 𝑀 3 : Cycle-Consistent Multi-Model Merging. arXiv preprint arXiv:2405.17897 (2024). — [arXiv:2405.17897](https://arxiv.org/abs/2405.17897) find
  33. Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. 2023. Seasoning model soups for robustness to adversarial and natural distribution shifts. In CVPR. 12313–12323. find
  34. Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2024. Model Merging by Uncertainty-Based Gradient Matching. In ICLR. find
  35. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 (2024). find
  36. Rui Dai, Sile Hu, Xu Shen, Yonggang Zhang, Xinmei Tian, and Jieping Ye. 2025. Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs. In ICLR. find
  37. MohammadReza Davari and Eugene Belilovsky. 2023. Model breadcrumbs: Scaling multi-task model merging with sparse masks. arXiv preprint arXiv:2312.06795 (2023). — [arXiv:2312.06795](https://arxiv.org/abs/2312.06795) find
  38. Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. 2024. DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling. arXiv preprint arXiv:2406.11617 (2024). — [arXiv:2406.11617](https://arxiv.org/abs/2406.11617) find
  39. Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. 2024. Controlled Text Generation via Language Model Arithmetic. ICLR (2024). find
  40. Caglar Demir, Arnab Sharma, and Axel-Cyrille Ngonga Ngomo. 2024. Adaptive Stochastic Weight Averaging. JMLR (2024). find
  41. Thomas G Dietterich et al. 2002. Ensemble learning. The handbook of brain theory and neural networks 2, 1 (2002), 110–125. find
  42. Omkar Dige, Diljot Singh, Tsz Fung Yau, Qixuan Zhang, Borna Bolandraftar, Xiaodan Zhu, and Faiza Khan Khattak. 2024. Mitigating Social Biases in Language Models through Unlearning. arXiv preprint arXiv:2406.13551 (2024). — [arXiv:2406.13551](https://arxiv.org/abs/2406.13551) find
  43. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220–235.
  44. Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. 2024. Avoiding Copyright Infringement via Machine Unlearning. arXiv preprint arXiv:2406.10952 (2024). — [arXiv:2406.10952](https://arxiv.org/abs/2406.10952) find
  45. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. 2018. Essentially no barriers in neural network energy landscape. In ICML. PMLR, 1309–1318. find
  46. Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Kin Tang, Daojing He, et al. 2024. Parameter competition balancing for model merging. NeurIPS 37 (2024), 84746–84776. find
  47. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2022. The role of permutation invariance in linear mode connectivity of neural networks. ICLR (2022). find
  48. Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. 2024. Proving linear mode connectivity of neural networks via optimal transport. In AISTATS. PMLR, 3853–3861. find
  49. Ronald A Fisher. 1922. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character 222, 594-604 (1922), 309–368. find
  50. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021. Sharpness-aware minimization for efficiently improving generalization. ICLR (2021). find
  51. Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, and Edouard Oyallon. 2024. WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average. arXiv preprint arXiv:2405.17517 (2024). — [arXiv:2405.17517](https://arxiv.org/abs/2405.17517) find
  52. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In ICML. PMLR, 3259–3269. find
  53. Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, and Rui Yan. 2024. Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction. ACL (2024). find
  54. Victor Gallego. 2024. Merging Improves Self-Critique Against Jailbreak Attacks. arXiv preprint arXiv:2406.07188 (2024). — [arXiv:2406.07188](https://arxiv.org/abs/2406.07188) find
  55. Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, and Murali Annavaram. 2024. Ethos: Rectifying language models in orthogonal parameter space. arXiv preprint arXiv:2403.08994 (2024). find
  56. Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. 2025. Task singular vectors: Reducing task interference in model merging. In CVPR. 18695–18705. find
  57. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. NeurIPS 31 (2018). find
  58. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s MergeKit: A Toolkit for Merging Large Language Models. arXiv preprint arXiv:2403.13257 (2024).
  59. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s MergeKit: A Toolkit for Merging Large Language Models. arXiv preprint arXiv:2403.13257 (2024).
  60. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025).
  61. Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. 2020. Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well. In ICLR. OpenReview.net. find
  62. Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. 2024. Model Merging and Safety Alignment: One Bad Model Spoils the Bunch. arXiv preprint arXiv:2406.14563 (2024). find
  63. Moritz Hardt, Ben Recht, and Yoram Singer. 2016. Train faster, generalize better: Stability of stochastic gradient descent. In ICML. PMLR, 1225–1234. find
  64. Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. 2024. Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations. arXiv preprint arXiv:2406.11801 (2024). find
  65. Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. 2023. MerA: Merging pretrained adapters for few-shot learning. arXiv preprint arXiv:2308.15982 (2023).
  66. Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. 2023. Merging Experts into One: Improving Computational Efficiency of Mixture of Experts. In EMNLP. find
  67. Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. 2024. Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic. Transactions on Machine Learning Research (2024). find
  68. Yifei He, Siqi Zeng, Yuzheng Hu, Rui Yang, Tong Zhang, and Han Zhao. 2025. MergeBench: A Benchmark for Merging Domain-Specialized LLMs. NeurIPS 2025 Datasets and Benchmarks Track (2025). find
  69. Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR (2019).
  70. Oğuz Kağan Hitit, Leander Girrbach, and Zeynep Akata. 2025. A Systematic Study of Model Merging Techniques in Large Language Models. arXiv preprint arXiv:2511.21437 (2025). — [arXiv:2511.21437](https://arxiv.org/abs/2511.21437) find
  71. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. NeurIPS 33 (2020), 6840–6851. find
  72. Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. 2019. The European Union general data protection regulation: what it is and what it means. Information & Communications Technology Law 28, 1 (2019), 65–98. find
  73. Stefan Horoi, Albert Manuel Orozco Camacho, Eugene Belilovsky, and Guy Wolf. 2024. Harmony in diversity: Merging neural networks with canonical correlation analysis. In ICML. find
  74. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR.
  75. Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang. 2024. Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. In AAAI, Vol. 38. 18252–18260. find
  76. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. COLM (2024).
  77. Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. 2024. EMR-Merging: Tuning-Free High-Performance Model Merging. arXiv preprint arXiv:2405.17461 (2024). — [arXiv:2405.17461](https://arxiv.org/abs/2405.17461) find
  78. Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, and Hung-yi Lee. 2024. Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities. ACL (2024). find
  79. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In ICLR.
  80. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022. Patching open-vocabulary models by interpolating weights. NeurIPS 35 (2022), 29262–29277. find
  81. Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. 2024. Transformer fusion with optimal transport. ICLR (2024). find
  82. Leonardo Iurada, Marco Ciccone, and Tatiana Tommasi. 2025. Efficient Model Editing with Task-Localized Sparse Fine-tuning. In ICLR. find
  83. P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. 2018. Averaging weights leads to wider optima and better generalization, In UAI. arXiv preprint arXiv:1803.05407, 876–885.
  84. Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. NeurIPS 31 (2018). find
  85. Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. 2018. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. JMLR 18, 223 (2018), 1–42. find
  86. Samyak Jain, Sravanti Addepalli, Pawan Kumar Sahu, Priyam Dey, and R Venkatesh Babu. 2023. Dart: Diversify-aggregate-repeat training improves generalization of neural networks. In CVPR. 16048–16059. find
  87. Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. 2024. Model Stock: All we need is just a few fine-tuned models. arXiv preprint arXiv:2403.19522 (2024). — [arXiv:2403.19522](https://arxiv.org/abs/2403.19522) find
  88. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564 (2023).
  89. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2023. Exploring the benefits of training expert language models over instruction tuning. In ICML. PMLR, 14702–14729.
  90. Divyansh Jhunjhunwala, Neharika Jali, Gauri Joshi, and Shiqiang Wang. 2024. Erasure Coded Neural Network Inference via Fisher Averaging. In ISIT. IEEE, 13–18. find
  91. Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. 2024. FedFisher: Leveraging Fisher Information for One-Shot Federated Learning. In AISTATS. PMLR, 1612–1620. find
  92. Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing Jiang, and Zi Huang. 2019. Learning private neural language modeling with attentive aggregation. In IJCNN. IEEE, 1–8. find
  93. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024). find
  94. Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, Jie Jiang, and Mingsheng Long. 2023. ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning. NeurIPS 36 (2023). find
  95. Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. 2024. Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task Arithmetic. arXiv:2407.07089 find
  96. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless Knowledge Fusion by Merging Weights of Language Models. In ICLR.
  97. Alexia Jolicoeur-Martineau, Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julien. 2024. Population parameter averaging (papa). TMLR (2024). find
  98. Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. 2023. Repair: Renormalizing permuted activations for interpolation repair. ICLR (2023). find
  99. Jean Kaddour. 2022. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. NeurIPS Workshop (2022). find
  100. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. 2022. When do flat minima optimizers work? NeurIPS 35 (2022), 16577–16595. find
  101. Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. 2024. Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts. arXiv preprint arXiv:2406.12034 (2024). — [arXiv:2406.12034](https://arxiv.org/abs/2406.12034) find
  102. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 (2019).
  103. Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166 (2023).
  104. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017… find
  105. Mateusz Klimaszewski, Piotr Andruszkiewicz, and Alexandra Birch. 2024. No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement. arXiv preprint arXiv:2404.15737 (2024). find
  106. Fanshuang Kong, Richong Zhang, and Ziqiao Wang. 2024. Activated Parameter Locating via Causal Intervention for Model Merging. arXiv preprint arXiv:2408.09485 (2024). — [arXiv:2408.09485](https://arxiv.org/abs/2408.09485) find
  107. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Generative Discriminator Guided Sequence Generation. EMNLP (2021).
  108. Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. 2019. Explaining landscape connectivity of low-cost solutions for multilayer nets. NeurIPS 32 (2019). find
  109. Yeoreum Lee, Jinwook Jung, and Sungyong Baik. 2025. Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning. In ICLR. find
  110. Bingdong Li, Zixiang Di, Yanting Yang, Hong Qian, Peng Yang, Hao Hao, Ke Tang, and Aimin Zhou. 2024. It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization. arXiv preprint arXiv:2407.00487 (2024). — [arXiv:2407.00487](https://arxiv.org/abs/2407.00487) find
  111. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. NeurIPS 31 (2018). find
  112. Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. 2023. Watermarking LLMs with Weight Quantization. arXiv preprint arXiv:2310.11237 (2023). find
  113. Lujun Li, Zhu Qiyuan, Jiacheng Wang, Wei Li, Hao Gu, Sirui Han, and Yike Guo. 2025. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. arXiv preprint arXiv:2506.23266 (2025). — [arXiv:2506.23266](https://arxiv.org/abs/2506.23266) find
  114. Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. 2024. MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation. arXiv:2406.07529 [cs.LG] — [arXiv:2406.07529](https://arxiv.org/abs/2406.07529) find
  115. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306 (2022).
  116. Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2024. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. ICLR (2024). find
  117. Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. 2023. Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions. In ICLR. find
  118. Tao Li, Weisen Jiang, Fanghui Liu, Xiaolin Huang, and James T Kwok. 2024. Scalable Learned Model Soup on a Single GPU: An Efficient Subspace Training Strategy. arXiv preprint arXiv:2407.03641 (2024). find
  119. Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37, 3 (2020), 50–60. find
  120. Wenyi Li, Huan-ang Gao, Mingju Gao, Beiwen Tian, Rong Zhi, and Hao Zhao. 2024. Training-Free Model Merging for Multi-target Domain Adaptation. arXiv preprint arXiv:2407.13771 (2024). — [arXiv:2407.13771](https://arxiv.org/abs/2407.13771) find
  121. Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. 2023. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698 (2023). find
  122. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. find
  123. Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, et al. 2025. Model Merging in Pre-training of Large Language Models. arXiv preprint arXiv:2505.12082 (2025). — [arXiv:2505.12082](https://arxiv.org/abs/2505.12082) find
  124. Tzu-Han Lin, Chen-An Li, Hung-yi Lee, and Yun-Nung Chen. 2024. DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging. arXiv preprint arXiv:2407.01470 (2024). find
  125. Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. 2022. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML. PMLR, 13857–13869. find
  126. Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. 2024. Checkpoint Merging via Bayesian Optimization in LLM Pretraining. arXiv preprint arXiv:2403.19390 (2024). — [arXiv:2403.19390](https://arxiv.org/abs/2403.19390) find
  127. Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Sergey Yekhanin, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better. arXiv preprint arXiv:2404.02241 (2024). — [arXiv:2404.02241](https://arxiv.org/abs/2404.02241) find
  128. Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, and Xia Hu. 2024. LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario. arXiv preprint arXiv:2403.00108 (2024). — [arXiv:2403.00108](https://arxiv.org/abs/2403.00108) find
  129. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. NeurIPS 35 (2022), 1950–1965.
  130. Tian Yu Liu, Aditya Golatkar, and Stefano Soatto. 2024. Tangent Transformers for Composition, Privacy and Removal. In ICLR. find
  131. Tian Yu Liu and Stefano Soatto. 2023. Tangent model composition for ensembling and continual fine-tuning. In ICCV. 18676–18686. find
  132. Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024. Towards Safer Large Language Models through Machine Unlearning. arXiv preprint arXiv:2402.10058 (2024).
  133. Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, and Chang Zhou. 2024. Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment. arXiv preprint arXiv:2405.17931 (2024). — [arXiv:2405.17931](https://arxiv.org/abs/2405.17931) find
  134. Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. 2024. Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. arXiv preprint arXiv:2406.15479 (2024). — [arXiv:2406.15479](https://arxiv.org/abs/2406.15479) find
  135. Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts. In SIGKDD. ACM, 1930–1939. find
  136. Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, Anton Protopopov, Alexis Audran-Reiss, Michael Slater, Despoina Magka, Tatiana Shavrina, Roberta Raileanu, et al. 2025. Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance. arXiv preprint arXiv:2511.13254 (2025). find
  137. Sébastien Marcel and Yann Rodriguez. 2010. Torchvision the machine-vision package of torch. In ACM MM. 1485–1488. find
  138. Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D Bagdanov, and Joost van de Weijer. 2025. No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. In ICML. find
  139. Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. 2024. MagMax: Leveraging Model Merging for Seamless Continual Learning. In ECCV. find
  140. Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. NeurIPS 35 (2022), 17703–17716.
  141. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In AISTATS. PMLR, 1273–1282. find
  142. Ruijie Miao, Yilun Yao, Zihan Wang, Zhiming Wang, Bairen Yi, LingJun Liu, Yikai Zhao, and Tong Yang. 2025. MergeMoE: Efficient Compression of MoE Models via Expert Output Merging. arXiv preprint arXiv:2510.14436 (2025). — [arXiv:2510.14436](https://arxiv.org/abs/2510.14436) find
  143. Adrian Robert Minut, Tommaso Mencattini, Andrea Santilli, Donato Crisostomi, and Emanuele Rodolà. 2025. Mergenetic: a Simple Evolutionary Model Merging Library. arXiv preprint arXiv:2505.11427 (2025). find
  144. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In ACL. Association for Computational Linguistics (ACL), 3470–3487. find
  145. Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. 2024. Learning to route among specialized experts for zero-shot generalization. ICML (2024). find
  146. Mohammed Muqeeth, Haokun Liu, and Colin Raffel. 2024. Soft merging of experts with adaptive routing. TMLR (2024). find
  147. Vaishnavh Nagarajan and J Zico Kolter. 2019. Uniform convergence may be unable to explain generalization in deep learning. NeurIPS 32 (2019). find
  148. Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, and Vishal M Patel. 2024. MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models. arXiv preprint arXiv:2404.09977 (2024). — [arXiv:2404.09977](https://arxiv.org/abs/2404.09977) find
  149. Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. 2024. Equivariant Deep Weight Space Alignment. ICML (2024). find
  150. Dang Nguyen, Trang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho. 2023. On cross-layer alignment for model fusion of heterogeneous neural networks. In ICASSP. IEEE, 1–5. find
  151. Dung V Nguyen, Anh T Nguyen, Minh H Nguyen, Luc Q Nguyen, Shiqi Jiang, Ethan Fetaya, Linh Duy Tran, Gal Chechik, and Tan M Nguyen. 2025. Expert Merging in Sparse Mixture of Experts with Nash Bargaining. arXiv preprint arXiv:2510.16138 (2025). — [arXiv:2510.16138](https://arxiv.org/abs/2510.16138) find
  152. Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. 2023. Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models. arXiv preprint arXiv:2311.08011 (2023). find
  153. Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS (2023). find
  154. Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. 2024. Towards modular llms by building and reusing a library of loras. ICML (2024). find
  155. Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. 2024. Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs. arXiv preprint arXiv:2406.16797 (2024). — [arXiv:2406.16797](https://arxiv.org/abs/2406.16797) find
  156. Stuart L Pardau. 2018. The california consumer privacy act: Towards a european-style privacy regime in the united states. J. Tech. L. & Pol’y 23 (2018), 68. find
  157. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In ICML. find
  158. Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. 2023. Re-basin via implicit Sinkhorn differentiation. In CVPR. 20237–20246. find
  159. Shiva Kumar Pentyala, Zhichao Wang, Bin Bi, Kiran Ramnath, Xiang-Bo Mao, Regunathan Radhakrishnan, Sitaram Asur, et al. 2024. PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning. arXiv preprint arXiv:2406.17923 (2024). — [arXiv:2406.17923](https://arxiv.org/abs/2406.17923) find
  160. Karol J Piczak. 2015. ESC: Dataset for environmental sound classification. In ACM MM. 1015–1018. find
  161. Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi, Simone Calderara, and Rita Cucchiara. 2024. A Second-Order perspective on Compositionality and Incremental Learning. arXiv preprint arXiv:2405.16350 (2024). — [arXiv:2405.16350](https://arxiv.org/abs/2405.16350) find
  162. Mohammad Areeb Qazi, Ibrahim Almakky, Anees Ur Rehman Hashmi, Santosh Sanjeev, and Mohammad Yaqub. 2024. DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Images. arXiv preprint arXiv:2404.14099 (2024). — [arXiv:2404.14099](https://arxiv.org/abs/2404.14099) find
  163. Haiquan Qiu, You Wu, Dong Li, Jianmin Guo, and Quanming Yao. 2025. Superpose Task-specific Features for Model Merging. In EMNLP. 4200–4214. find
  164. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. find
  165. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. NeurIPS 36 (2023).
  166. Dhananjay Ram, Aditya Rawal, Momchil Hardalov, Nikolaos Pappas, and Sheng Zha. 2024. DEM: Distribution Edited Model for Training with Mixed Data Distributions. arXiv preprint arXiv:2406.15570 (2024). find
  167. Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. 2023. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In ICML. PMLR, 28656–28679. find
  168. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. NeurIPS 36 (2023).
  169. Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. 2024. WARP: On the Benefits of Weight Averaged Rewarded Policies. arXiv preprint arXiv:2406.16768 (2024). — [arXiv:2406.16768](https://arxiv.org/abs/2406.16768) find
  170. Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. 2022. Diverse weight averaging for out-of-distribution generalization. NeurIPS 35 (2022), 10821–10836. find
  171. Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models. ICML (2024). find
  172. Sylvestre-Alvise Rebuffi, Francesco Croce, and Sven Gowal. 2023. Revisiting adapters with adversarial training. ICLR. find
  173. Anthony Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science (1995). find
  174. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In CVPR. 10684–10695.
  175. Simo Ryu. [n. d.]. Merging loras. In https://github.com/ cloneofsimo/lora. ↗
  176. Omer Sagi and Lior Rokach. 2018. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery 8, 4 (2018), e1249. find
  177. Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, et al. 2023. Early Weight Averaging meets High Learning Rates for LLM Pre-training. In NeurIPS Workshop. find
  178. Raphael Schumann, Elman Mansimov, Yi-An Lai, Nikolaos Pappas, Xibin Gao, and Yi Zhang. 2024. Backward Compatibility During Data Updates by Weight Interpolation. In EACL. 2846–2861. find
  179. Ozan Sener and Vladlen Koltun. 2018. Multi-Task Learning as Multi-Objective Optimization. In NeurIPS. 525–536. find
  180. Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. 2023. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311.13600 (2023). — [arXiv:2311.13600](https://arxiv.org/abs/2311.13600) find
  181. Ken Shoemake. 1985. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques. 245–254. find
  182. Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. 2023. UnIVAL: Unified Model for Image, Video, Audio and Language Tasks. TMLR (2023). find
  183. Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. 2021. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In ICML. PMLR, 9722–9732. find
  184. Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. NeurIPS 33 (2020), 22045–22055. find
  185. Ronald Skorobogat, Karsten Roth, and Mariana-Iuliana Georgescu. 2025. Subspace-boosted model merging. arXiv preprint arXiv:2506.16506 (2025). — [arXiv:2506.16506](https://arxiv.org/abs/2506.16506) find
  186. Joshua Smith and Michael Gashler. 2017. An investigation of how neural networks learn from the experiences of peers through periodic weight averaging. In 2017 16th IEEE ICML and Applications (ICMLA). IEEE, 731–736. find
  187. Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. In ICML. 32211–32252. find
  188. George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. 2024. ZipIt! Merging Models from Different Tasks without Training. ICLR (2024). find
  189. George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. 2024. Model merging with SVD to tie the Knots. In ICLR. find
  190. Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. 2024. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM. arXiv preprint arXiv:2403.07816 (2024). — [arXiv:2403.07816](https://arxiv.org/abs/2403.07816) find
  191. Wenju Sun, Qingyong Li, Wen Wang, Yang Liu, Yangli-ao Geng, and Boyang Li. 2025. Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration. In NeurIPS. find
  192. Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. 2020. Adashare: Learning what to share for efficient deep multi-task learning. NeurIPS 33 (2020), 8728–8740. find
  193. Anirudh S Sundar, Chao-Han Huck Yang, David M Chan, Shalini Ghosh, Venkatesh Ravichandran, and Phani Sankar Nidadavolu. 2024. Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification. ICASSP Workshop (2024). find
  194. Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. 2023. An Empirical Study of Multimodal Model Merging. EMNLP (2023). find
  195. Derek Tam, Mohit Bansal, and Colin Raffel. 2023. Merging by Matching Models in Task Subspaces. arXiv preprint arXiv:2312.04339 (2023). find
  196. Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, and Colin Raffel. 2024. Realistic evaluation of model merging for compositional generalization. arXiv preprint arXiv:2409.18314 (2024). find
  197. Derek Tam, Margaret Li, Prateek Yadav, Rickard Brüel Gabrielsson, Jiacheng Zhu, Kristjan Greenewald, Mikhail Yurochkin, Mohit Bansal, Colin Raffel, and Leshem Choshen. 2024. Llm merging: Building llms efficiently through merging. In NeurIPS 2024 Competition Track. find
  198. Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. 2023. Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion. arXiv preprint arXiv:2312.06173 (2023). — [arXiv:2312.06173](https://arxiv.org/abs/2312.06173) find
  199. Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, and Bo Du. 2024. Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion. arXiv preprint arXiv:2406.09770 (2024). — [arXiv:2406.09770](https://arxiv.org/abs/2406.09770) find
  200. Anke Tang, Li Shen, Yong Luo, Enneng Yang, Han Hu, Leifei Zhang, Bo Do, and Dacheng Tao. 2025. FusionBench: A Comprehensive Benchmark of Deep Model Fusion. In Journal of Machine Learning Research. find
  201. Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. 2024. Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML (2024). find
  202. Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. 2024. Parameter Efficient Multi-task Model Fusion with Partial Linearization. ICLR (2024). find
  203. Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Lefei Zhang, Bo Du, and Dacheng Tao. 2025. Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging. In NeurIPS. find
  204. Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, and Yansong Feng. 2024. Unlocking the Potential of Model Merging for Low-Resource Languages. arXiv preprint arXiv:2407.03994 (2024). find
  205. Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS 30 (2017). find
  206. Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. 2020. Optimizing mode connectivity via neuron alignment. NeurIPS 33 (2020), 15300–15311. find
  207. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023).
  208. Kimi Team. 2025. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 [cs.AI] https://arxiv.org/abs/2501.12599 ↗
  209. Qwen Team. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  210. Ayush Thakur and Rashmi Vashisth. 2024. A unified module for accelerating stable-diffusion: Lcm-lora. arXiv preprint arXiv:2403.16024 (2024). find
  211. DK Thennal, Ganesh Nathan, and MS Suchithra. 2024. Fisher Mask Nodes for Language Model Merging. In LREC-COLING. 7349–7355. find
  212. Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. 2022. Unrolling sgd: Understanding factors influencing machine unlearning. In EuroS&P. IEEE, 303–319. find
  213. Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. 2025. WSM: decay-free learning rate schedule via checkpoint merging for LLM pre-training. arXiv preprint arXiv:2507.17634 (2025). — [arXiv:2507.17634](https://arxiv.org/abs/2507.17634) find
  214. Jie Tian, Xiaoye Qu, Zhenyi Lu, Wei Wei, Sichen Liu, and Yu Cheng. 2025. Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think. In CVPR. 12512–12521. find
  215. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  216. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  217. Joachim Utans. 1996. Weight averaging for neural networks and local resampling schemes. In AAAI Workshop. Citeseer, 133–138. find
  218. Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2021. Multi-task learning for dense prediction tasks: A survey. TPAMI 44, 7 (2021), 3614–3633. find
  219. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS 30 (2017).
  220. Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans, Christian Henning, Benjamin F. Grewe, and João Sacramento. 2021. Neural networks with late-phase weights. In ICLR. find
  221. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. [n. d.]. Knowledge Fusion of Large Language Models. In The Twelfth International Conference on Learning Representations.
  222. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. ICLR (2024).
  223. Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. 2024. FuseChat: Knowledge Fusion of Chat Models. arXiv preprint arXiv:2402.16107 (2024).
  224. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235 (2023).
  225. Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. 2024. LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks. ACL (2024). find
  226. Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. 2020. Federated Learning with Matched Averaging. In ICLR. find
  227. Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. NeurIPS 33 (2020), 7611–7623. find
  228. Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. 2025. LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging. In ICLR. find
  229. Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. 2024. Localizing Task Information for Improved Model Merging and Compression. ICML (2024). find
  230. Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, et al. 2024. Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning. arXiv preprint arXiv:2407.15762 (2024). — [arXiv:2407.15762](https://arxiv.org/abs/2407.15762) find
  231. Peng Wang, Li Shen, Zerui Tao, Shuaida He, and Dacheng Tao. 2024. Generalization Analysis of Stochastic Weight Averaging with General Sampling. In ICML. find
  232. Xiang Wang, Annie N Wang, Mo Zhou, and Rong Ge. 2022. Plateau in Monotonic Linear Interpolation—A" Biased" View of Loss Landscape for Deep Networks. In ICLR. find
  233. Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, and Hongxia Yang. 2025. Model Merging Scaling Laws in Large Language Models. arXiv preprint arXiv:2509.24244 (2025). — [arXiv:2509.24244](https://arxiv.org/abs/2509.24244) find
  234. Zijing Wang, Xingle Xu, Yongkang Liu, Yiqun Zhang, Peiqin Lin, Shi Feng, Xiaocui Yang, Daling Wang, and Hinrich Schütze. 2025. Why Do More Experts Fail? A Theoretical Analysis of Model Merging. arXiv preprint arXiv:2505.21226 (2025).
  235. Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. 2023. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218 (2023). — [arXiv:2307.09218](https://arxiv.org/abs/2307.09218) find
  236. Qi Wei, Shuo He, Enneng Yang, Tingcong Liu, Haobo Wang, Lei Feng, and Bo An. 2025. Representation Surgery in Model Merging with Probabilistic Modeling. In ICML. find
  237. Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang, Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. 2025. Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging. arXiv preprint arXiv:2505.19892 (2025). — [arXiv:2505.19892](https://arxiv.org/abs/2505.19892) find
  238. Yongxian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. 2025. Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent. In ICML. find
  239. Ross Wightman. 2019. PyTorch Image Models. https://github.com/rwightman/pytorch-image-models. https://doi.org/10.5281/zenodo.4414861 — [doi:10.5281/zenodo.4414861](https://doi.org/10.5281/zenodo.4414861) ↗
  240. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019).
  241. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In IC…
  242. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. 2022. Robust fine-tuning of zero-shot models. In CVPR. 7959–7971. find
  243. Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. 2022. Backdoor-Bench: A Comprehensive Benchmark of Backdoor Learning. In NeurIPS Datasets and Benchmarks Track. find
  244. Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. 2023. pi-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation. In ICML. PMLR, 37713–37727. find
  245. Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, and Mingxuan Yuan. 2025. Unlocking efficient long-to-short llm reasoning with model merging. arXiv preprint arXiv:2503.20641 (2025). — [arXiv:2503.20641](https://arxiv.org/abs/2503.20641) find
  246. Taiqiang Wu, Runming Yang, Tao Liu, Jiahao Wang, and Ngai Wong. 2025. Revisiting Model Interpolation for Efficient Reasoning. arXiv preprint arXiv:2510.10977 (2025).
  247. Xun Wu, Shaohan Huang, and Furu Wei. 2024. MoLE: Mixture of LoRA Experts. In ICLR. find
  248. Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. 2024. Lm-cocktail: Resilient tuning of language models via model merging. ACL (2024). find
  249. Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. 2024. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.18729 (2024). — [arXiv:2411.18729](https://arxiv.org/abs/2411.18729) find
  250. Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024. Instructional fingerprinting of large language models. arXiv preprint arXiv:2401.12255 (2024). find
  251. Zhengqi Xu, Ke Yuan, Huiqiong Wang, Yong Wang, Mingli Song, and Jie Song. 2024. Training-Free Pretrained Model Merging. CVPR (2024). find
  252. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. Resolving Interference When Merging Models. NeurIPS (2023). find
  253. Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. 2024. What Matters for Model Merging at Scale? arXiv preprint arXiv:2410.03617 (2024). find
  254. Kunda Yan, Min Zhang, Sen Cui, Qu Zikun, Bo Jiang, Feng Liu, and Changshui Zhang. 2025. CALM: Consensus-Aware Localized Merging for Multi-Task Learning. In ICML. find
  255. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 (2023).
  256. Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. 2025. Kwai keye-vl 1.5 technical report. arXiv preprint arXiv:2509.01563 (2025). — [arXiv:2509.01563](https://arxiv.org/abs/2509.01563) find
  257. Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. 2023. AdaTask: A task-aware adaptive learning rate approach to multi-task learning. In AAAI, Vol. 37. 10745–10753. find
  258. Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. 2024. Representation Surgery for Multi-Task Model Merging. ICML (2024). find
  259. Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xingwei Wang, Xiaocun Cao, Jie Zhang, and Dacheng Tao. 2024. Surgeryv2: Bridging the gap between model merging and multi-task learning with deep representation surgery. arXiv preprint arXiv:2410.14389 (2024). — [arXiv:2410.14389](https://arxiv.org/abs/2410.14389) find
  260. Enneng Yang, Anke Tang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, and Jie Zhang. 2025. Continual Model Merging without Data: Dual Projections for Balancing Stability and Plasticity. NeurIPS (2025). find
  261. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024. AdaMerging: Adaptive Model Merging for Multi-Task Learning. ICLR (2024).
  262. Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. 2019. SWALP: Stochastic weight averaging in low precision training. In ICML. PMLR, 7015–7024. find
  263. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large Language Model Unlearning. In Socially Responsible Language Modelling Research.
  264. Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, and Wanli Ouyang. 2025. Dynamic model merging with mixture of weights. TCSVT (2025). find
  265. Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, Yuning Zhang, and Wanli Ouyang. 2023. Merging Vision Transformers from Different Tasks and Domains. arXiv preprint arXiv:2312.16240 (2023). — [arXiv:2312.16240](https://arxiv.org/abs/2312.16240) find
  266. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP. In EMNLP. 7163–7189.
  267. Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. 2024. A safety realignment framework via subspace-oriented model fusion for large language models. arXiv preprint arXiv:2405.09055 (2024). — [arXiv:2405.09055](https://arxiv.org/abs/2405.09055) find
  268. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement. arXiv preprint arXiv:2408.03092 (2024). — [arXiv:2408.03092](https://arxiv.org/abs/2408.03092) find
  269. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML (2024).
  270. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. NeurIPS 33 (2020), 5824–5836.
  271. Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. 2019. Bayesian nonparametric federated learning of neural networks. In ICML. PMLR, 7252–7261. find
  272. Kerem Zaman, Leshem Choshen, and Shashank Srivastava. 2023. Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion. arXiv preprint arXiv:2311.07682 (2023). find
  273. Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. 2025. Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion. In ICML. find
  274. Frederic Z Zhang, Paul Albert, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. 2024. Knowledge Composition using Task Vectors with Learned Anisotropic Scaling. arXiv preprint arXiv:2407.02880 (2024). — [arXiv:2407.02880](https://arxiv.org/abs/2407.02880) find
  275. Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf. 2020. Swa object detection. arXiv preprint arXiv:2012.12645 (2020). — [arXiv:2012.12645](https://arxiv.org/abs/2012.12645) find
  276. Haobo Zhang and Jiayu Zhou. 2025. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging. arXiv preprint arXiv:2505.22934 (2025). find
  277. Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. 2023. Composing Parameter-Efficient Modules with Arithmetic Operations. In NeurIPS. find
  278. Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. 2024. BadMerging: Backdoor Attacks Against Model Merging. CCS (2024). find
  279. Jiangtao Zhang, Shunyu Liu, Jie Song, Tongtian Zhu, Zhengqi Xu, and Mingli Song. 2023. Lookaround Optimizer: 𝑘 steps around, 1 step average. NeurIPS 36 (2023). find
  280. Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. 2025. Understanding Mode Connectivity via Parameter Space Symmetry. In ICML. find
  281. Xinyu Zhao, Guoheng Sun, Ruisi Cai, Yukun Zhou, Pingzhi Li, Peihao Wang, Bowen Tan, Yexiao He, Li Chen, Yi Liang, et al. 2024. Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild. NeurIPS 37 (2024), 13349–13371. find
  282. Yiran Zhao, Wenxuan Zhang, Huiming Wang, Kenji Kawaguchi, and Lidong Bing. 2024. AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging. arXiv preprint arXiv:2402.18913 (2024). find
  283. Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. 2024. LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild. ACL (2024). find
  284. Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. 2024. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792 (2024). — [arXiv:2404.16792](https://arxiv.org/abs/2404.16792) find
  285. Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. 2023. Learn from model beyond fine-tuning: A survey. arXiv preprint arXiv:2310.08184 (2023). — [arXiv:2310.08184](https://arxiv.org/abs/2310.08184) find
  286. Luca Zhou, Daniele Solombrino, Donato Crisostomi, Maria Sofia Bucarelli, Giuseppe Alessio D’Inverno, Fabrizio Silvestri, and Emanuele Rodolà. 2025. On Task Vectors and Gradients. arXiv preprint arXiv:2508.16082 (2025). — [arXiv:2508.16082](https://arxiv.org/abs/2508.16082) find
  287. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic. arXiv preprint arXiv:2406.11385 (2024). find
  288. Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. 2024. Cross-Task Linearity Emerges in the Pretraining-Finetuning Paradigm. arXiv preprint arXiv:2402.03660v1 (2024). — [arXiv:2402.03660](https://arxiv.org/abs/2402.03660) find
  289. Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. 2023. Going beyond linear mode connectivity: The layerwise linear feature connectivity. NeurIPS 36 (2023), 60853–60877. find
  290. Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. 2024. Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models. ICML (2024). find
  291. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-Tuning Language Models from Human Preferences. arXiv preprint arXiv:1909.08593 (2019). find
  292. Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. 2024. Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging. In ICLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2024-model-merging-llms-mllms-methods/