Elamur Research⌕
☰
home / papers / 2022 / Dataless Knowledge Fusion by Merging Weights of Language Models

Dataless Knowledge Fusion by Merging Weights of Language Models

Paper · 2022 · Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro et al.

Summary

Many fine-tuned language models are published while the labeled data behind them stays private, which rules out multi-task learning as a way to combine what they know. Jin et al. study how to fuse several such models, all fine-tuned from one shared pretrained checkpoint, into a single network of unchanged size. The merged network should work across all source domains and transfer to unseen ones. Their method, Regression Mean (RegMean), treats every linear layer as a least-squares problem. It picks merged weights whose outputs stay close, in ℓ2 terms, to each individual model's outputs on that model's own layer inputs. This gives a closed-form answer. Contributors only need to share Gram matrices of layer inputs (X^T X), never raw examples. In experiments with RoBERTa, T5 and DeBERTa on several dataset collections, RegMean consistently beats Simple averaging, Fisher-weighted averaging and model ensembling. On out-of-domain data it can even exceed the best single model. Because merging requires no retraining, it offers an inexpensive, privacy-preserving substitute for multi-task training.

Key points

  • Merging formula: W_M = (Σ X_iᵀX_i)⁻¹ Σ X_iᵀX_i W_i, applied independently to each linear layer. Diagonal entries of the Gram matrices reweight rows of the weight matrices, while off-diagonal entries mix them. When all Gram matrices are identical, the formula collapses to plain averaging.
  • Efficiency: each model's input statistics are computed once and can be reused for any combination of models. Merging therefore scales to many models without access to the training data.
  • Regularization: shrinking the off-diagonal entries by relative scaling worked clearly better than adding a constant to the diagonal. This was tested by merging T5-base emotion classification models and measuring average in-domain F1. The authors attribute the gap to input scales that differ across layers, models and datasets.
  • Model choice: out-of-domain results depend strongly on which models are merged. Among RoBERTa-base emotion models, combining the TEC and ISEAR models, which were the two strongest individual OOD performers, gave the best OOD score.
  • Permutation matching: weight-based matching from OTFusion found no permutation (it returned the identity), which the authors link to shared initialization and residual connections. Activation-based matching from Git Re-Basin did produce permutations but performed far worse than simple averaging. Code is available at github.com/bloomberg/dataless-model-merging.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States175 works · 37,050 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng. 2022. Dataless Knowledge Fusion by Merging Weights of Language Models. arXiv:2212.09849.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.09849 · arXiv 2212.09849

@misc{jin2022dataless,
  title = {{Dataless Knowledge Fusion by Merging Weights of Language Models}},
  author = {Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng},
  year = {2022},
  doi = {10.48550/arxiv.2212.09849},
  eprint = {2212.09849},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.09849}
}

References

57 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836, 2022. — [arXiv:2209.04836](https://arxiv.org/abs/2209.04836) find
  2. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of human language technology conference and conference on empirical methods in natural language processing, pp. 579–586, 2005.
  3. Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. arXiv preprint arXiv:2110.10832, 2021. — [arXiv:2110.10832](https://arxiv.org/abs/2110.10832) find
  4. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017.
  5. Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405–22418, 2021. find
  6. Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. ArXiv, abs/2204.03044, 2022. — [arXiv:2204.03044](https://arxiv.org/abs/2204.03044) find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  8. Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
  9. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309– 1318. PMLR, 2018. find
  10. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259–3269. PMLR, 2020. find
  11. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1–9, 2007.
  12. Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. International Conference on Learning Representations, 2020. find
  13. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021.
  14. Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pp. 57–60, 2006. find
  15. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018.
  16. Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging language models using pre-trained distillation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2874–2887, 2021. find
  17. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306, 2022.
  18. Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019. find
  19. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957, 2017.
  20. Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. FedNLP: Benchmarking federated learning methods for natural language processing tasks. In Findings of the Association for Computational Linguistics:… find
  21. Vicki Liu, Carmen Banea, and Rada Mihalcea. Grounded emotions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 477–483. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  23. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. NIPS Workshop on Learning with Limited Data, 2017. find
  24. Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. arXiv preprint arXiv:2111.09832, 2021.
  25. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017. find
  26. Saif Mohammad. \# emotional tweets. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 246–255, 2012…
  27. Saif Mohammad and Felipe Bravo-Marquez. Wassa-2017 shared task on emotion intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 34–49, 2017.
  28. Saif M Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4):480–499, 2015. find
  29. Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning, pp. 4743–4751. PMLR, 2019. find
  30. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 512–523, 2020. find
  31. Dang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho. Model fusion of heterogeneous neural networks via cross-layer alignment. arXiv preprint arXiv:2110.15538, 2021. — [arXiv:2110.15538](https://arxiv.org/abs/2110.15538) find
  32. Laura Ana Maria Oberländer and Roman Klinger. An analysis of annotated corpora for emotion classification in text. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 2104–2119, 2018. find
  33. David Opitz and Richard Maclin. Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11:169–198, 1999. find
  34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Sou… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  35. Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018.
  36. Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10585–10605, Online and Punta Cana, Dominican Republic, November 2021.
  37. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Associati…
  38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
  39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, 2016.
  40. Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. ArXiv, abs/2205.09739, 2022. — [arXiv:2205.09739](https://arxiv.org/abs/2205.09739) find
  41. Shruti Rijhwani and Daniel Preotiuc-Pietro. Temporally-informed analysis of named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7605–7617, Online, July 2020. find
  42. Lior Rokach. Ensemble-based classifiers. Artificial intelligence review, 33(1):1–39, 2010. find
  43. Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147, 2003.
  44. Klaus R Scherer and Harald G Wallbott. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2):310, 1994. find
  45. Hendrik Schuff, Jeremy Barnes, Julian Mohme, Sebastian Padó, and Roman Klinger. Annotation, modelling and analysis of fine-grained emotions on a stance and sentiment detection corpus. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis…
  46. Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055, 2020. find
  47. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2…
  48. Carlo Strapparava and Rada Mihalcea. Semeval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pp. 70–74, 2007.
  49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
  50. Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020a. find
  51. Jing Wang, Mayank Kulkarni, and Daniel Preoţiuc-Pietro. Multi-domain named entity recognition with genre-aware and agnostic inference. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 8476–8488, 2020b. find
  52. Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205.12410, 2022. — [arXiv:2205.12410](https://arxiv.org/abs/2205.12410) find
  53. Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641, 2019. find
  54. Orion Weller, Kevin Seppi, and Matt Gardner. When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 272–282, Dublin, Ireland, May 2… find
  55. Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018.
  56. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
  57. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Internat…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2022-dataless-knowledge-fusion-merging-weights/