Elamur Research
☰
home / papers / 2020 / Revisiting Few-sample BERT Fine-tuning

Revisiting Few-sample BERT Fine-tuning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2006.05987

Summary

Fine-tuning BERT on small datasets is notoriously unreliable. Runs that differ only in random seed can produce very different or even collapsed models, so practitioners often train many restarts and keep the best one. This study traces the instability to three widespread practices. First, BERTADAM, the default optimizer, skips Adam's bias-correction step. Second, the uppermost pre-trained layers make a poor starting point for downstream tasks. Third, fine-tuning typically runs for a short, fixed schedule. The authors test each factor on GLUE tasks in few-sample settings and evaluate on held-out test splits they carved out themselves. They propose three fixes: restoring debiased Adam, re-initializing the top layers, and training for longer. With these changes, degenerate runs disappear and results vary far less across seeds, so mass random restarts are no longer needed. The fixes partly overlap; for example, longer training can make up for the biased optimizer. Earlier stabilization techniques show much smaller benefits under this corrected recipe. The work offers practical guidance for anyone fine-tuning large pre-trained encoders on limited labeled data.

Key points

  • Optimizer issue. BERTADAM leaves out Adam's bias-correction lines. The authors argue this distorts the effective learning rate early in training, which matters most for small datasets that only train for a few iterations.
  • Library spread. The biased optimizer was inherited by the official BERT code, huggingface's Transformers, AllenNLP, GluonNLP, jiant, MT-DNN and FARM. Transformers switched its default to debiased Adam in July 2019.
  • Data. Experiments use eight GLUE datasets, with a focus on RTE, MRPC, STS-B and CoLA (under 10k training examples each). All datasets were also downsampled to 1k examples. Original validation sets were split to create test data.
  • Setup. The model is uncased 24-layer BERT Large, trained with batch size 32, peak learning rate 2×10⁻⁵, 3 epochs by default, and 20 random seeds per configuration.
  • Re-evaluated methods. The study revisits intermediate-task training (Phang et al.), regularization toward pre-trained weights (Lee et al.), and techniques from Howard & Ruder. All three help much less once the improved procedure is used.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

154 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 148 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.05987 · arXiv 2006.05987

@misc{zhang2020revisiting,
  title = {{Revisiting Few-sample BERT Fine-tuning}},
  author = {Zhang, Tianyi and Wu, Felix and Katiyar, Arzoo and Weinberger, Kilian Q. and Artzi, Yoav},
  year = {2020},
  doi = {10.48550/arxiv.2006.05987},
  eprint = {2006.05987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.05987}
}

References

61 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC 2009 Workshop, 2009. find
  2. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017, 2017. find
  3. Ciprian Chelba and Alex Acero. Adaptation of maximum entropy capitalizer: Little data can help a lot. In EMNLP, 2004. find
  4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020. find
  5. Hal Daumé III. Frustratingly easy domain adaptation. In ACL, 2007. find
  6. Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning to initialize. In NeurIPS, 2019. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. find
  8. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. find
  9. William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In IWP, 2005. find our card
  10. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. Allennlp: A deep semantic natural language processing platform. In NLP-OSS, 2018. find our card
  11. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010. find
  12. Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, and Shuai Zheng. Gluoncv and gluonnlp: Deep learning in computer vision and natural language processing. arXiv preprint arXiv:1907.04… — [arXiv:1907.04433](https://arxiv.org/abs/1907.04433) find
  13. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. find
  15. J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In EMNLP, 2019. find
  16. John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In NAACL, 2019. find our card
  17. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019. find
  18. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018. find
  19. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs. https://tinyurl.com/y2y8u5ed, 2017. ↗
  20. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  21. Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In NeurIPS, 1992. find
  22. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. find
  23. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020. find
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. find our card
  25. Xingjian Li, Haoyi Xiong, Haozhe An, Chengzhong Xu, and Dejing Dou. Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer. In ICML, 2020. find
  26. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855, 2019a. find our card
  27. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019b. find our card
  28. Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, and Jianfeng Gao. The microsoft toolkit of multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:2002.07972, 2020. find
  29. Yang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318, 2019. — [arXiv:1903.10318](https://arxiv.org/abs/1903.10318) find
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692, 2019c. find
  31. Brian W Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 1975. find
  32. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to bert embeddings during fine-tuning? arXiv preprint arXiv:2004.14448, 2020. find
  33. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884, 2020. find
  34. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR, 2019. find
  35. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987, 2019. find
  36. Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. find our card
  37. Martin Popel and Ondřej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 2018. find
  38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. find
  40. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014. find
  41. Asa Cooper Stickland and Iain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In ICML, 2019. find
  42. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In CCL, 2019. find our card
  43. Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferability in pretrained language models. In EMNLP, 2020. find
  44. Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In ACL, 2019a. find our card
  45. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR, 2019b. find our card
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. find
  47. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In EMNLP-IJCNLP, 2019. find our card
  48. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML, 2013. find
  49. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019a. find our card
  50. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019b. find our card
  51. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phil Yeres, Jason Phang, Haokun Liu, Phu Mon Htut, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Kath… find
  52. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. TACL, 2019. find
  53. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In ACL, 2018. find
  54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. find our card
  55. Wei Yang, Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document retrieval. arXiv preprint arXiv:1903.10972, 2019a. — [arXiv:1903.10972](https://arxiv.org/abs/1903.10972) find
  56. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019b. find our card
  57. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014. find
  58. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In ICLR, 2019. find
  59. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. In ICLR, 2020. find
  60. Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. Incorporating bert into neural machine translation. In ICLR, 2020. find
  61. RTE NLI MRPC Paraphrase STS-B Similarity Task \# of training samples \# of validation samples \# of test samples 2.5k 139 139 3.7k 204 205 Evaluation metric Acc. Majority baseline (val) 52.9 Majority baseline (test) 52.5 F1 81.3 81.2 Table 3: The datasets used in this work. We apply non-standard data s… find