Elamur Research
☰
home / papers / 2021 / BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2106.10199

Summary

Fine-tuning a whole BERT-style masked language model for each downstream task produces a separate large copy per task. That is expensive to store and deploy, and it hides what fine-tuning actually changed. The authors propose BitFit (BIas-Term FIne-Tuning). It freezes every weight matrix and layer-norm gain of the pre-trained encoder and trains only the additive bias vectors plus a task-specific linear classifier. They test it with BERT-BASE, BERT-LARGE and RoBERTa-BASE (HuggingFace implementations) on the GLUE benchmark, PTB POS-tagging and growing subsets of SQuAD v1.0. The baselines are full fine-tuning, Diff-Pruning and Adapters. BitFit updates under 0.1% of the parameters. On small and medium datasets it roughly equals full fine-tuning and sometimes beats it, and with large datasets it stays competitive with other sparse methods. Every task changes the same small set of parameters, so one shared backbone can serve many tasks. The results also suggest that fine-tuning mostly brings out knowledge learned in pre-training rather than teaching new linguistic skills.

Key points

  • Bias terms make up only 0.09% of BERT-BASE parameters and 0.08% of BERT-LARGE. Average GLUE dev scores with BitFit vs. full fine-tuning: 82.4 vs. 82.3 (BERT-BASE), 84.2 vs. 84.1 (BERT-LARGE) and 84.6 vs. 85.3 (RoBERTa-BASE).
  • On BERT-LARGE, BitFit trains about 6x fewer parameters than Diff-Pruning (0.5%) and 45x fewer than Adapters (3.6%). It beats Diff-Pruning on 4 of 9 dev-set tasks and does especially well on small tasks such as RTE (73.2 vs. 71.9 for full fine-tuning).
  • Training only the query bias (b_q) and the intermediate MLP bias (b_m2) uses 0.04% of parameters and averages 81.1 on BERT-BASE. The key bias barely moves during fine-tuning.
  • Bias terms matter specifically: tuning a random parameter subset of the same size scores 78.5 (uniform sampling) or 79.5 (whole rows/columns), and a fully frozen model scores 62.1.
  • On PTB POS-tagging, BitFit matches full fine-tuning (97.2/97.4/97.1 vs. 97.2/97.4/97.2). On SQuAD, BitFit leads when training data is small, and full fine-tuning pulls ahead as data grows. BitFit also has a smaller generalization gap. Code: github.com/benzakenelad/BitFit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

377 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 356 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.10199 · arXiv 2106.10199

@misc{benzaken2021bitfit,
  title = {{BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models}},
  author = {Ben-Zaken, Elad and Ravfogel, Shauli and Goldberg, Yoav},
  year = {2021},
  doi = {10.48550/arxiv.2106.10199},
  eprint = {2106.10199},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.10199}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Gethsiyal Augasta and T. Kathirvalavakumar. 2013. Pruning algorithms of neural networks - a comparative study. Central Eur. J. Comput. Sci., 3(3):105– 115. find
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. find
  3. Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541. find
  4. Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. Tiny transfer learning: Towards memory-efficient on-device learning. CoRR, abs/2007.11622. — [arXiv:2007.11622](https://arxiv.org/abs/2007.11622) find
  5. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. find
  6. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIP… find
  7. Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. 2020. Multi-head attention: Collaborate instead of concatenate. CoRR, abs/2006.16362. — [arXiv:2006.16362](https://arxiv.org/abs/2006.16362) find
  8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  10. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  11. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  12. Jonathan Frankle, David J. Schwab, and Ari S. Morcos. 2020. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. CoRR, abs/2003.00152. — [arXiv:2003.00152](https://arxiv.org/abs/2003.00152) find
  13. Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. CoRR, abs/2002.08307. find
  14. Demi Guo, Alexander M. Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. find our card
  15. Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28:1135–1143. find
  16. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531. find our card
  17. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. CoRR, abs/1902.00751. find
  18. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs. find
  19. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. find our card
  20. Ehud D. Karnin. 1990. A simple procedure for pruning back-propagation trained neural networks. IEEE Trans. Neural Networks, 1(2):239–242. find
  21. Chao Liu, Zhiyong Zhang, and Dong Wang. 2014. Pruning deep neural networks by optimal brain damage. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014, pages 1092–1095. ISCA. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  23. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. find
  24. Fixing CoRR, Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 312–318. Assoc… find
  25. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net… find
  26. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. find
  27. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3208–3229. Association for Computational Linguistics. find
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250. find our card
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  30. Russell Reed. 1993. Pruning algorithms-a survey. IEEE Trans. Neural Networks, 4(5):740–747. find
  31. Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press. find
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  33. Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Özlem Aslan, Shengjie Wang, Abdelrahman Mohamed, Matthai Philipose, Matthew Richardson, and Rich Caruana. 2017. Do deep convolutional nets really need to be deep and convolutional? In 5th International Conference on Learning Representations,… find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. find
  35. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461. find our card
  36. Shengjie Wang, Tianyi Zhou, and Jeff A. Bilmes. 2019. Bias also matters: Bias attribution for deep neural network explanation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning… find
  37. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. find
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin L… find our card
  39. Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226–2241, Online. Association for Compu… find