Elamur Research⌕
☰
home / papers / 2020 / AdapterHub: A Framework for Adapting Transformers

AdapterHub: A Framework for Adapting Transformers

Summary

Fine-tuning large pre-trained transformers separately for every task produces a full model copy each time. These copies are costly to store, slow to share and hard to combine. Adapters address this. They are small bottleneck modules added to each layer and trained while the base model stays frozen. In practice, though, they were seldom released on their own, and their designs differed in subtle ways. AdapterHub, built on HuggingFace Transformers, standardizes how adapters are trained, saved, uploaded and loaded for models such as BERT, RoBERTa and XLM-R. Its configuration files express the known adapter architecture variants. Its public website organizes shared adapters by task, language and dataset. Training an adapter takes about two extra lines in an existing script, and loading a published one takes a single line. On GLUE with BERT-Base, two adapter designs score close to full fine-tuning while needing only megabytes per task. This makes multi-task, cross-lingual and low-resource transfer research cheaper, more reproducible and easier to build on.

Key points

  • GLUE comparison (BERT-Base, mean of 3 runs, bottleneck 48):
    • Full fine-tuning was compared with the Pfeiffer et al. (2020a) adapter and the Houlsby et al. (2019) adapter.
    • The scores are broadly comparable. For example, on RTE the scores are 66.2 (full), 70.8 (Pfeiffer) and 69.8 (Houlsby). On MNLI all three reach 84.1.
  • Adapter size (Pfeiffer design):
    • At a compression rate of 64, an adapter adds about 0.2M parameters (0.9Mb) on a Base model and 0.8M (3.2Mb) on a Large model.
    • At a rate of 2, it grows to 7.1M parameters (28Mb) on Base and 25.2M (97Mb) on Large.
    • The Houlsby variant needs roughly twice the space.
  • Storage comparison:
    • BERT-Base is about 440Mb, and XLM-R Large is about 2.2Gb compressed.
    • The disk space of two fully fine-tuned BERT-Base copies can hold about 125 task adapters.
    • Over 99% of the parameters stay shared and frozen.
  • Framework features:
    • Adapter weights are extracted and saved separately from the base model.
    • Prediction heads can optionally be uploaded alongside the adapter.
    • Stacking and composition of adapters are supported out of the box.
  • Website and compatibility:
    • The AdapterHub.ml site browses adapters through three levels: task or language, then task category or training language, then dataset or domain (e.g., SST, Swahili Wikipedia).
    • Adapters only work with the base model they were trained on.

Interesting ideas

  • I1. Describe architecture variants in a declarative config. Placement choices and extra LayerNorms are set in configuration rather than in code. This lets one codebase reproduce many published variants and lets users fairly compare their own variants, which suits any family of small design tweaks.
  • I2. Ship the delta, not the whole model. Saving only the trained adapter keeps per-task artifacts tiny. This makes it practical to maintain many task- or language-specific models, for example for Ukrainian tasks, on limited storage or mobile devices.
  • I3. Exploit frozen-backbone compatibility. Each adapter must produce output that fits the frozen next layer. Because of this, independently trained task and language adapters can be stacked or swapped. One could train a language adapter once and reuse it across several task adapters.
  • I4. Organize shared models hierarchically. Indexing models by task or language, then by subcategory, then by dataset or domain, and filtering by base model, makes a shared library easy to search. The same structure could organize the Elamur library's own models and datasets.
  • I5. Keep integration cost to a line or two. Requiring minimal changes to existing training scripts lowers the barrier to adoption. This is worth copying when introducing any internal tool to collaborators.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

286 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 269 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-demos.7

@inproceedings{pfeiffer2020adapterhub,
  title = {{AdapterHub: A Framework for Adapting Transformers}},
  author = {Pfeiffer, Jonas and Rücklé, Andreas and Poth, Clifton and Kamath, Aishwarya and Vulić, Ivan and Ruder, Sebastian and Cho, Kyunghyun and Gurevych, Iryna},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.7},
  url = {https://doi.org/10.18653/v1/2020.emnlp-demos.7}
}

References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, U… find our card
  2. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint. find our card
  3. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. find our card
  4. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Nov… find our card
  5. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the PASCAL@ACL 2007. find
  6. Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10… find
  7. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the PASCAL@ACL 2006. find
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find our card
  9. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Ass… find our card
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lo… find
  11. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328–339. find
  12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint. find
  13. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs [online]. 2017. find
  14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. find our card
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv preprint. find
  16. Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2019. Specializing unsupervised pretraining models for word-level semantic similarity. arXiv preprint. find
  17. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 506–516. find our card
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint. find
  19. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. AdapterDrop: On the Efficiency of Adapters in Transformers. arXiv preprint. find our card
  20. Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, B… find
  21. Sebastian Ruder. 2017. An Overview of Multi-Task Learning in Deep Neural Networks. arXiv preprint. find our card
  22. Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapol… find our card
  23. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, Florence, Italy, August 2, 2019, pages 7–14. find
  24. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterFusion: Non-destructive task composition for transfer learning. arXiv preprint. find our card
  25. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Virtual Conference. find our card
  26. Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, T… find
  27. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint. find our card
  28. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. find
  29. June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. find
  30. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. find
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi an-dArt Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020. Hugging-Face’s Transformers: State-of-the-art Natural Language Processing. In Proceedings of the 2020 Conference on Empirica… find our card
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 20… find
  33. Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 5986–5995. find
  34. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3645– 3650. find our card
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… find
  36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels,… find our card
  37. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint. find our card
  38. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  39. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/