Elamur Research⌕
☰
home / papers / 2023 / AdaMerging: Adaptive Model Merging for Multi-Task Learning

AdaMerging: Adaptive Model Merging for Multi-Task Learning

Paper · 2023 · Enneng Yang, Zhenyi Wang, Li Shen et al.

Summary

Merging several fine-tuned copies of one pre-trained model into a single multi-task network is attractive when the original training data are unavailable. Task-vector methods such as Task Arithmetic and Ties-Merging do this, but they lag well behind jointly trained MTL and depend heavily on a hand-picked scaling coefficient. The authors propose AdaMerging, which learns the merging weights automatically and without labels. It has two variants: one weight per task vector (task-wise) or one per layer of each task vector (layer-wise). The weights are tuned by minimizing the Shannon entropy of the merged model's predictions on unlabeled test inputs. This idea comes from test-time adaptation, and the authors support it by showing that entropy correlates positively with prediction loss. Experiments merge CLIP-style ViT models (ViT-B/32, ViT-B/16, ViT-L/14) fine-tuned on eight image classification datasets. AdaMerging clearly outperforms earlier merging schemes, generalizes better to unseen tasks and is more robust to distribution shift. This makes data-free model merging a more practical alternative to retraining.

Key points

  • The eight benchmark tasks are SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST and DTD. On ViT-B/32, Task Arithmetic scores 69.1% average accuracy and Ties-Merging 72.9%, both at their best λ = 0.3. Layer-wise AdaMerging reaches 80.1%, and the variant AdaMerging++ reaches 81.1%. Traditional joint MTL remains the upper bound at 88.9%.
  • The method adds very few trainable parameters. Task-wise AdaMerging learns 8 coefficients and layer-wise learns 1,248, while the merged task vectors total about 907.6M parameters. Layer-wise tuning took roughly 125 extra minutes on ViT-B/32.
  • Gains over Task Arithmetic and Ties-Merging are about 5–11% on ViT-B/32 and 4.4–9.1% on unseen downstream tasks. Across seven test-time distribution shifts, the average improvement over Task Arithmetic is 8.45%.
  • Spearman correlation between entropy and loss is positive on all eight datasets and throughout optimization. This supports entropy as an unsupervised surrogate objective.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓University of Maryland, College ParkCollege Park · United States140 works · 29,011 citations🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Northeastern UniversityShenyang · China53 works · 2,205 citations🏢JingdongBeijing · China4 works · 929 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang and Dacheng Tao. 2023. AdaMerging: Adaptive Model Merging for Multi-Task Learning. arXiv:2310.02575.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.02575 · arXiv 2310.02575

@misc{yang2023adamerging,
  title = {{AdaMerging: Adaptive Model Merging for Multi-Task Learning}},
  author = {Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang and Dacheng Tao},
  year = {2023},
  doi = {10.48550/arxiv.2310.02575},
  eprint = {2310.02575},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.02575}
}

References

71 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023. find
  2. Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997. find
  3. Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NeurIPS, 34:22405–22418, 2021. find
  4. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pp. 794–803. PMLR, 2018. find
  5. Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In NeurIPS, 2020. find
  6. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017. find
  7. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pp. 3606–3613, 2014. find
  8. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, pp. 160–167, 2008. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT 2019, pp. 4171–4186. Association for Computational Linguistics, 2019.
  10. Ke Ding, Xin Dong, Yong He, Lei Cheng, Chilin Fu, Zhaoxin Huan, Hai Li, Tan Yan, Liang Zhang, Xiaolu Zhang, et al. Mssm: a multiple-level sparse sharing model for efficient multi-task learning. In SIGIR, pp. 2237–2241, 2021. find
  11. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In ACL, pp. 1723–1732, 2015.
  12. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
  13. Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594-604):309–368, 1922. find
  14. Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. NeurIPS, 17, 2004. find
  15. Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In ICML, pp. 3854–3863. PMLR, 2020. find
  16. Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In ICLR. OpenReview.net, 2020. find
  17. Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. NeurIPS, 34:29335–29347, 2021. find
  18. Yun He, Xue Feng, Cheng Cheng, Geng Ji, Yunsong Guo, and James Caverlee. Metabalance: Improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. WWW, pp. 2205–2215, 2022. find
  19. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. find
  20. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
  21. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
  22. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023.
  23. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
  24. Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, Jie Jiang, and Mingsheng Long. Forkmerge: Mitigating negative transfer in auxiliary-task learning. NeurIPS, 36, 2024. find
  25. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
  26. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  27. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pp. 554–561, 2013. find
  28. Yann LeCun. The mnist database of handwritten digits. http://yann.lecun. com/exdb/mnist/, 1998. ↗
  29. Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023. find
  30. Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In ICLR, 2019. find
  31. Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361, 2023. — [arXiv:2303.15361](https://arxiv.org/abs/2303.15361) find
  32. Petro Liashchynskyi and Pavlo Liashchynskyi. Grid search, random search, genetic algorithm: a big comparison for nas. arXiv preprint arXiv:1912.06059, 2019. — [arXiv:1912.06059](https://arxiv.org/abs/1912.06059) find
  33. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. NeurIPS, 34:18878–18890, 2021. find
  34. Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML, pp. 13857–13869. PMLR, 2022. find
  35. Jialin Liu, Antoine Moreau, Mike Preuss, Jeremy Rapin, Baptiste Roziere, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 620–628, 2020. find
  36. Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In CVPR, pp. 1871–1880. Computer Vision Foundation / IEEE, 2019a. find
  37. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Anna Korhonen, David R. Traum, and Lluı́s Màrquez (eds.), ACL, pp. 4487–4496. Association for Computational Linguistics, 2019b.
  38. Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, pp. 5334–5343, 2017. find
  39. Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In SIGKDD, pp. 1930–1939. find
  40. Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. NeurIPS, 35:17703–17716, 2022.
  41. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, pp. 3994–4003. IEEE Computer Society, 2016. find
  42. Leann Myers and Maria J Sirois. Spearman correlation coefficients, differences between. Encyclopedia of statistical sciences, 12, 2004. find
  43. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, pp. 16888–16905. PMLR, 2022. find
  44. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023. find
  45. Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurIPS, 2023. find
  46. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. find
  47. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10): 1872–1897, 2020.
  48. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748–8763. PMLR, 2021.
  49. Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Arno Solin. Uncertainty-guided source-free domain adaptation. In ECCV, pp. 537–555. Springer, 2022. find
  50. Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NeurIPS, pp. 525–536, 2018. find
  51. Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. find
  52. Derun Song, Enneng Yang, Guibing Guo, Li Shen, Linying Jiang, and Xingwei Wang. Multi-scenario and multi-task aware feature interaction for recommendation system. TKDD, 2024. find
  53. Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, pp. 1453–1460. IEEE, 2011. find
  54. Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, pp. 9120–9132. PMLR, 2020. find
  55. George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.03053, 2023. — [arXiv:2305.03053](https://arxiv.org/abs/2305.03053) find
  56. Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. NeurIPS, 33:8728–8740, 2020. find
  57. Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. TPAMI, 44(7): 3614–3633, 2021. find
  58. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021. find
  59. Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020. find
  60. Zhenyi Wang, Xiaoyang Wang, Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. Meta-learning without data via wasserstein distributionally-robust model fusion. In UAI, pp. 2045–2055. PMLR, 2022. find
  61. Zhenyi Wang, Li Shen, Tiehang Duan, Qiuling Suo, Le Fang, Wei Liu, and Mingchen Gao. Distributionally robust memory evolution with generalized divergence for continual learning. TPAMI, 2023. find
  62. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, pp…
  63. Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. pi-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. In ICML, pp. 37713–37727. PMLR, 2023. find
  64. Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In ICLR. OpenReview.net, 2020. find
  65. Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. IJCV, 119:3–22, 2016. find
  66. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. NeurIPS, 2023. find
  67. Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In AAAI, volume 37, pp. 10745–10753, 2023. find
  68. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NeurIPS, 27, 2014. find
  69. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. NeurIPS, 33:5824–5836, 2020.
  70. Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. find
  71. Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870, 2023. — [arXiv:2306.14870](https://arxiv.org/abs/2306.14870) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2023-adamerging-adaptive-model-merging-multi/