Elamur Research⌕
☰
home / papers / 2026 / DivMerge: A divergence-based model merging method for multi-tasking

DivMerge: A divergence-based model merging method for multi-tasking

Paper · 2026 · Brahim Touayouch, Loïc Fosse, Géraldine Damnati et al.

Summary

Merging several fine-tuned models into one multi-task model costs far less than joint multi-task training. However, tasks tend to interfere with each other, and this gets worse as more models are combined. A central obstacle is that merging has no explicit target distribution to optimize toward.

The authors (Orange Research and academic partners) propose DivMerge, which uses each fine-tuned model's output distribution as a substitute target. The method works with task vectors, meaning the parameter shifts from a shared base model, inside the Task Arithmetic framework. It iteratively learns merging coefficients, either one per task or one per layer. The learning objective is to reduce the Jensen-Shannon divergence between the merged model and every fine-tuned model, measured on unlabelled inputs from each task, so no reference labels are needed. The paper also links the approach theoretically to weight disentanglement and to conventional dataset-mixing multi-task learning.

When tested with autoregressive models on both classification and generation tasks, DivMerge outperforms earlier merging methods and loses less performance as the number of tasks grows. This makes it a label-free way to reuse specialized models shared publicly, for example on Hugging Face.

Key points

  • Baselines include Model Averaging, Multi-SLERP, TIES and AdaMerging, an entropy-based optimization method. DivMerge shows narrower confidence intervals than AdaMerging, indicating more stable results.
  • Results are reported with the ANP metric on classification pairs such as CoLA+SST-2, QNLI+MNLI and RTE+MRPC, and on mixed classification/generation sets with increasing task counts (95% CIs).
  • DivMerge beats the non-optimization baselines with only 25 unlabelled samples per task. This is about 0.4% of the fine-tuning training data and 5% of the validation set. Optimization converges smoothly without signs of overfitting.
  • A gap to joint training remains: standard multi-task learning reaches an ANP of 0.98 in the classification setup.
  • Stated limitations: only full fine-tuning was tested (no LoRA), and the method assumes access to samples from each task's true input distribution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓École PolytechniquePalaiseau · France14 works · 898 citations🎓Aix-Marseille UniversitéMarseille · France24 works · 39,160 citations🎓Institut Polytechnique de ParisPalaiseau · France15 works · 1,612 citations🔬Laboratoire d’Informatique et SystèmesMarseille · France8 works · 106 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Brahim Touayouch, Loïc Fosse, Géraldine Damnati and Gwénolé Lecorvé. 2026. DivMerge: A divergence-based model merging method for multi-tasking. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.337

@inproceedings{touayouch2026divmerge,
  title = {{DivMerge: A divergence-based model merging method for multi-tasking}},
  author = {Brahim Touayouch, Loïc Fosse, Géraldine Damnati and Gwénolé Lecorvé},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.337},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.337}
}

References

71 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shun-ichi Amari. 2006. Differential geometry of statistical inference. In Proc. USSR-Japan Symposium. find
  2. Shun-Ichi Amari. 2009. a-divergence is unique, belonging to both f-divergence and bregman divergence classes. IEEE Transactions on Information Theory, 55(11):4925–4931. find
  3. Shun-ichi Amari and Hiroshi Nagaoka. 2000. Methods of information geometry. American Mathematical Soc. find
  4. Dana H Ballard. 1987. Modular learning in neural networks. In Proc. of the National conference on Artificial intelligence. find
  5. Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. 2005. Clustering with bregman divergences. Journal of machine learning research, 6(Oct):1705–1749. find
  6. Jonathan Baxter. 2000. A model of inductive bias learning. Journal of artificial intelligence research, 12:149–198. find
  7. Shai Ben-David and Reba Schuller. 2003. Exploiting task relatedness for multiple task learning. In Proc. Annual Conference on Learning Theory. find
  8. Yaiza Bermudez, Gaetan Bisson, Iñaki Esnaola, and Samir Perlaza. 2025. Démonstrations des théorèmes folkloriques sur la dérivée de radon-nikodym. In GRETSI’25-XXXe Colloque Francophone de Traitement du Signal et des Images. find
  9. Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proc. of the IEEE conference on computer vision and pattern recognition. find
  10. Yochai Blau and Tomer Michaeli. 2019. Rethinking lossy compression: The rate-distortion-perception tradeoff. In Proc. ICML. find
  11. Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. 2020. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In Proc. ECCV. find
  12. Lev M Bregman. 1967. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217. find
  13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Proc. NEURIPS.
  14. Rich Caruana. 1997. Multitask Learning. Machine learning, 28(1):41–75. find
  15. Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2024. Language and task arithmetic with parameter-efficient layers for zero-shot summarization. In Proc. of the Fourth Workshop on Multilingual Representation Learning. find
  16. T. M. Cover and J. A. Thomas. 1991. Elements of Information Theory. John Wiley & Sons, Inc. find
  17. H. Crauel. 2002. Random Probability Measures on Polish Spaces. Taylor & Francis. find
  18. Imre Csiszár. 1975. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146–158. find
  19. Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2023. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310.12808. — [arXiv:2310.12808](https://arxiv.org/abs/2310.12808) find
  20. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems. find
  21. Olivier Ferret. 2025. Projeter pour mieux fusionner: une histoire de bandit et de lit. In Proc. TALN. find
  22. Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently Identifying Task Groupings for Multi-Task Learning. In Proc. NEURIPS. find
  23. Philippe Formont, Maxime Darrin, Banafsheh Karim-ian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, and Pablo Piantanida. 2025. Learning task-agnostic representations through multi-teacher distillation. arXiv preprint arXiv:2510.18680. find
  24. Loïc Fosse, Frederic Bechet, Benoit Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, and Pablo Piantanida. 2025. Statistical deficiency for task inclusion estimation. In Proc. ACL. find
  25. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257.
  26. Karsten Grove and Hermann Karcher. 1973. How to conjugate c 1-close group actions. Mathematische Zeitschrift. find
  27. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  28. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. In Proc. ICLR.
  29. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In Proc. ICLR.
  30. Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. 2016. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1–32. find
  31. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407.
  32. Frank Nielsen. 2020. On a generalization of the jensen– shannon divergence and the jensen–shannon centroid. Entropy, 22(2):221. find
  33. Young Kyun Jang, Dat Huynh, Ashish Shah, Wen-Kai Chen, and Ser-Nam Lim. 2024. Spherical linear interpolation and text-anchoring for zero-shot composed image retrieval. In Proc. ECCV. find
  34. Wooseong Jeong and Kuk-Jin Yoon. 2025. Selective task group updates for multi-task optimization. arXiv preprint arXiv:2502.11986. — [arXiv:2502.11986](https://arxiv.org/abs/2502.11986) find
  35. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In Proc. ICLR. find
  36. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2. find
  37. Hyoseo Kim, Dongyoon Han, and Junsuk Choe. 2024. Negmerge: Consensual weight negation for strong machine unlearning. arXiv preprint arXiv:2410.05583. — [arXiv:2410.05583](https://arxiv.org/abs/2410.05583) find
  38. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  39. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86. find
  40. Kevin Kuo, Amrith Setlur, Kartik Srinivas, Aditi Raghunathan, and Virginia Smith. 2025. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.04626. — [arXiv:2504.04626](https://arxiv.org/abs/2504.04626) find
  41. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Proc. Findings of EMNLP. find
  42. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proc. of Text Summarization Branches Out, Barcelona, Spain.
  43. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proc. ACL.
  44. Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. In Proc. NEURIPS.
  45. Frank Nielsen and Richard Nock. 2007. On the centroids of symmetrized bregman divergences. arXiv preprint arXiv:0711.3242. — [arXiv:0711.3242](https://arxiv.org/abs/0711.3242) find
  46. Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Proc. NEURIPS. find
  47. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proc. EACL.
  48. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023. Modular deep learning. arXiv preprint arXiv:2302.11529. — [arXiv:2302.11529](https://arxiv.org/abs/2302.11529) find
  49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. EMNLP.
  51. Henry Scheffé. 1947. A Useful Convergence Theorem for Probability Distributions. The Annals of Mathematical Statistics. find
  52. Zhang Shengyu, Dong Linfeng, Li Xiaoya, Zhang Sen, Sun Xiaofei, Wang Shuhe, Li Jiwei, Runyi Hu, Zhang Tianwei, Fei Wu, and 1 others. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
  53. Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020. Which tasks should be learned together in multi-task learning? In Proc. ICLR. find
  54. Charles Stein. 1956. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proc. of the third Berkeley symposium on mathematical statistics and probability. find
  55. Gilbert W Stewart. 1993. On the early history of the singular value decomposition. SIAM review. find
  56. Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699. — [arXiv:1910.10699](https://arxiv.org/abs/1910.10699) find
  57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Proc. NEURIPS.
  58. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proc. ICML.
  59. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. EMNLP.
  60. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. In Proc. NEURIPS.
  61. Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813. — [arXiv:2405.07813](https://arxiv.org/abs/2405.07813) find
  62. Yuefeng Zhang. 2023. A rate-distortion-classification approach for lossy image compression. Digital Signal Processing, 141:104163. find
  63. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  64. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. Metagpt: Merging large language models using model exclusive task arithmetic. In Proc. EMNLP. find
  65. Andrew KC Wong and Manlai You. 1985. Entropy and distance of random graphs with application to structural pattern recognition. IEEE transactions on pattern analysis and machine intelligence, (5):599– 609. find
  66. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and 1 others. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time…
  67. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. In Proc. NEURIPS.
  68. Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. 2024. What matters for model merging at scale? arXiv preprint arXiv:2410.03617. find
  69. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024a. Qwen2.5 technical report. arXiv preprint arX… find
  70. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024b. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666. find
  71. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024c. Adamerging: Adaptive model merging for multi-task learning. In Proc. ICLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/touayouch-2026-divmerge-divergence-based-model-merging/