Elamur Research⌕
☰
home / papers / 2023 / MerA: Merging Pretrained Adapters For Few-Shot Learning

MerA: Merging Pretrained Adapters For Few-Shot Learning

Paper · 2023 · Shwai He, Run-Ze Fan, Liang Ding et al.

Summary

Adapter tuning trains only small bottleneck modules inside a frozen pretrained language model. These modules start from random weights, so they tend to underperform when only a handful of labeled examples are available. AdapterFusion reuses previously trained adapters, but its attention-style composition layers add many trainable parameters and require keeping several adapters at inference. A pilot study in this paper even shows a single adapter beating AdapterFusion on 10-shot GLUE tasks. The authors therefore propose MerA, which combines several pretrained adapters from AdapterHub into one adapter before few-shot tuning. Besides plain summation and averaging, MerA can first align neurons across adapters with optimal transport, based on either weights or activations. On GLUE with BERT and RoBERTa, the activation-based variant consistently beats standard adapters and AdapterFusion at the same small parameter cost. Merging adapters from the same NLP track as the target task gives the largest gains. The approach offers a cheap way to transfer adapter knowledge in low-data settings.

Key points

  • Source adapters come from AdapterHub and were trained on imdb, boolq, scitail and winogrande, covering sentiment, QA, NLI and commonsense reasoning. Evaluation uses GLUE with 10, 20 and 30 training shots.
  • Four merging methods are compared: "Sum", "Avg", "Wts" and "Acts". The two optimal-transport alignments beat the naive ones. "Acts" is the default, with up to 2.7% average improvement over a standard adapter.
  • Trainable parameters: MerA and a plain adapter use about 0.8% of the model, AdapterFusion about 18%. In 30-shot runs averaged over head, prompt and prompt-with-demonstration tuning, MerA scores 71.1 vs 69.2 for a standard adapter on BERT-Base, and 78.9 vs 76.4 on RoBERTa-Base. Both MerA averages also exceed full fine-tuning.
  • Under the "same-track" setting, MerA beats the standard adapter by 3.5% on MRPC (using STS adapters) and 5.0% on MNLI (using NLI adapters). Adapters from mismatched tracks can hurt accuracy.
  • In a zero-shot check, plugging MerA into BERT gives a better starting point than random adapters or no adapters, for example +2.4% on MNLI and +0.98% on SST-2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States141 works · 29,174 citations🎓The University of SydneySydney · Australia49 works · 68,350 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou and Dacheng Tao. 2023. MerA: Merging Pretrained Adapters For Few-Shot Learning. arXiv:2308.15982.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.15982 · arXiv 2308.15982

@misc{he2023mera,
  title = {{MerA: Merging Pretrained Adapters For Few-Shot Learning}},
  author = {Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou and Dacheng Tao},
  year = {2023},
  doi = {10.48550/arxiv.2308.15982},
  eprint = {2308.15982},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.15982}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee, and Andrew McCallum. 2022. Meta-adapters: Parameter efficient few-shot fine-tuning through meta-learning. In AutoML. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
  3. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  5. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint. find
  6. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL.
  7. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022a. Towards a unified view of parameter-efficient transfer learning. In ICLR.
  8. Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, and Dacheng Tao. 2023. Pad-net: An efficient framework for dynamic networks. In ACL. find
  9. Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022b. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. In EMNLP. find
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML. find
  11. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI. find
  12. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  13. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Others. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint.
  15. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL.
  16. Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. 2022. Adaptable adapters. In NAACL. find
  17. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In EACL.
  18. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In EMNLP Demonstrations.
  19. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint.
  20. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM. find
  21. Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. In NeurIPS. find
  22. Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas J. Guibas. 2015. Convolutional wasserstein distances: efficient optimal transportation on geometric domains. ACM Trans. Graph. find
  23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
  24. Zhenyi Wang, Xiaoyang Wang, Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. 2022. Meta-learning without data via wasserstein distributionally-robust model fusion. In UAI. find
  25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
  26. Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022. On the complementarity between pre-training and random-initialization for resource-rich machine translation. In COLING. find
  27. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint. find
  28. Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. 2023. Revisiting token dropping strategy in efficient BERT pretraining. In ACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2023-mera-merging-pretrained-adapters-few/