Elamur Research⌕
☰
home / papers / 2024 / AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

Paper · 2024 · Han Zhou, Xingchen Wan, Ivan Vulić et al.

Summary

Parameter-efficient fine-tuning (PEFT) adapts large pretrained language models by training only a small fraction of weights. Choosing a PEFT setup is still mostly manual, though: which module to use, how large to make it and which Transformer layers should receive it. Such hand-picked choices are likely to be suboptimal. AutoPEFT addresses this with automated search, borrowing ideas from neural architecture search. The authors build a search space from three building blocks: prefix-tuning, plus adapters inserted either sequentially or side by side. Each has a searchable size, and each layer can be switched on or off. Multi-objective Bayesian optimization then explores this space, using cheap one-epoch training runs as a proxy. It returns a Pareto front of configurations that balance accuracy against the number of trainable parameters. Configurations found on a single task transfer well to other tasks. On GLUE and SuperGLUE, with both encoder-only and encoder-decoder models, they beat existing PEFT baselines and match or exceed full-model fine-tuning. This matters because a modest one-time search yields reusable, efficient setups.

Key points

  • The search space combines three module types: Prefix-Tuning (PT), together with adapters placed in sequence (Serial Adapters, SA) or alongside the layer (Parallel Adapters, PA):
    • Each module's size (bottleneck dimension or prefix length) is chosen on a binary-logarithmic scale from 0 up to the hidden size.
    • A size of 0 means the module is absent from that layer.
  • The space is very large: about 5.45 million configurations for BERT base and roughly 3×10¹⁰ for RoBERTa large.
  • Search cost is kept low by training each candidate for only one epoch. Searching on a small, fast task and then transferring the result also helps.
  • In a layer-selection comparison on RTE with BERT large, AutoPEFT outperformed three baselines while using 12.5% fewer parameters:
    • AutoPEFT: 10 active layers, bottleneck 128, score 73.86.
    • Serial adapter: 72.56.
    • AdapterDrop: 73.36.
    • Adaptable Adapter: 73.50.
  • Code is released at github.com/cambridgeltl/autopeft. The framework is designed so that new PEFT modules can be added later.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Han Zhou, Xingchen Wan, Ivan Vulić and Anna Korhonen. 2024. AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00662

@article{zhou2024autopeft,
  title = {{AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning}},
  author = {Han Zhou, Xingchen Wan, Ivan Vulić and Anna Korhonen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00662},
  url = {https://doi.org/10.1162/tacl_a_00662}
}

References

53 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, Dublin, Ireland. Association for Computatio… find
  2. Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine-tuning design spaces. In The Eleventh International Conference on Learning Representations.
  3. Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. 2022b. Adaptformer: Adapting vision transformers for scalable visual recognition. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIP… find
  4. Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G. Wilson, and Eytan Bakshy. 2020. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in Neural Information Processing Systems, 33:21524–21538. find
  5. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Asso…
  6. Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. 2021. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Dece… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla understanding. In Proceedings of the 2019 Conference of the North American Chapt…
  8. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25–29, 2022.
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Lon… find
  10. Xuanyi Dong and Yi Yang. 2020. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. find
  11. Edward J. Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022a. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  12. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017. https://doi.org/10.1007/978-3-030-05318-5 11 — [doi:10.1007/978-3-030-05318-5](https://doi.org/10.1007/978-3-030-05318-5) ↗
  13. David Eriksson, Pierce I-Jen Chuang, Samuel Daulton, Peng Xia, Akshat Shrivastava, Arun Babu, Shicong Zhao, Ahmed A. Aly, Ganesh Venkatesh, and Maximilian Balandat. 2021. Latency-aware neural architecture search with multi-objective bayesian optimization. In 8th ICML Workshop on Automated Machine Le… find
  14. Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. 2022b. Sparse structure search for delta tuning. In Advances in Neural Information Processing Systems. find
  15. Sergio Izquierdo, Julia Guerrero-Viu, Sven Hauns, Guilherme Miotto, Simon Schrodi, André Biedenkapp, Thomas Elsken, Difan Deng, Marius Lindauer, and Frank Hutter. 2021. Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization. In 8th ICML Workshop on Aut… find
  16. David Eriksson and Martin Jankowiak. 2021. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence, pages 493–503. PMLR. find
  17. Peter I. Frazier. 2018. A tutorial on bayesian optimization. CoRR, abs/1807.02811v1. https://doi.org/10.48550/arXiv.1807.02811 — [doi:10.48550/arxiv.1807.02811](https://doi.org/10.48550/arxiv.1807.02811) ↗
  18. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
  19. Roman Garnett. 2023. Bayesian Optimization. Cambridge University Press. https://doi.org/10.1017/9781108348973 — [doi:10.1017/9781108348973](https://doi.org/10.1017/9781108348973) ↗
  20. Demi Guo, Alexander Rush, and Yoon Kim. 2021 Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4…
  21. Liam Li and Ameet Talwalkar. 2019. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22–25, 2019, pages 367–377. find
  22. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio… — [doi:10.18653/v1/2021](https://doi.org/10.18653/v1/2021) find
  23. Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019a. DARTS: Differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. find
  24. Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems. ciation for Computational Linguistics: Human Languag…
  25. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
  26. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstration…
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692v1.
  28. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Ponti. 2023. Modular deep learning. Transactions on Machine Learning Research. Survey Certification. https://doi.org/10.48550/arXiv.2302.11529 — [doi:10.48550/arxiv.2302.11529](https://doi.org/10.48550/arxiv.2302.11529) ↗
  29. Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pages 1… find
  30. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673. Association for Computational Linguist…
  31. Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:140:1–140:67.
  33. Nafise Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. 2022. Adaptable adapters. In Proceedings of the 2022 Conference of the North American Chapter of the Asso-Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. 2021. A comprehensive surve… find
  34. Neural Information Processing Systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pages 24193–24205. find
  35. Bin Xin Ru, Xingchen Wan, Xiaowen Dong, and Michael A. Osborne. 2021. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3–7, 2021. find
  36. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1 /P19-14…
  37. Robin Ru, Pedro M. Esperança, and Fabio Maria Carlucci. 2020. Neural architecture generator optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6–12, 2020, virtual. find
  38. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 32…
  39. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930–7946, Online and Punta…
  40. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240. Association for Computational Lingu…
  41. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  42. Xingchen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A. Osborne. 2021. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In International Conference on Machine Learning, pages 10663–10674. PMLR. find
  43. Xingchen Wan, Binxin Ru, Pedro M. Esperança, and Zhenguo Li. 2022. On redundancy and diversity in cell-based neural architecture search. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25–29, 2022. OpenReview.net. find
  44. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference Yi-Lin Sung…
  45. Antoine Yang, Pedro M. Esperança, and Fabio Maria Carlucci. 2020. NAS evaluation is frustratingly hard. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. find
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  47. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations. find
  48. Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226–2241. Association for Computationa… find
  49. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. AdaMix: Mixture-of-adaptations for parameter-efficient model tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5744–5760, A…
  50. Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. 2023. Survival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13064–13077, Singapore. Association for Computational… find
  51. Colin White, Willie Neiswanger, and Yash Savani. 2021. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2… find
  52. Eckart Zitzler and Lothar Thiele. 1998. Multiobjective optimization using evolutionary algorithms - A comparative case study. In Parallel Problem Solving from Nature - PPSN V, 5th International Conference, Amsterdam, The Netherlands, September 27–30, 1998, Proceedings, volume 1498 of Lecture Notes i… — [doi:10.1007/bfb0056872](https://doi.org/10.1007/bfb0056872) find
  53. Saining Xie, Alexander Kirillov, Ross B. Girshick, and Kaiming He. 2019. Exploring randomly wired neural networks for image recognition. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, Barret Zoph and Quoc V. Le. 2017. Neu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2024-autopeft-automatic-configuration-search-parameter/