Elamur Research
☰
home / papers / 2021 / Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.568.pdf

Summary

The authors explain why huge pretrained language models with hundreds of millions of parameters can be fine-tuned with ordinary gradient descent on a few hundred or a few thousand labeled examples without overfitting. To answer this, they use the notion of intrinsic dimension. This is the smallest number of free parameters whose optimization in a random subspace yields 90% of the performance of full fine-tuning (the d90 threshold). Parameters from the small subspace are mapped into the full weight space using the Fastfood transform. The authors also propose a variant, SAID, in which a separate scale is learned for each layer. The experiments cover GLUE tasks and other datasets and more than a dozen models: BERT, RoBERTa, T5, XLM-R, Albert, XLNet, ELECTRA. They show that hundreds or thousands of parameters are usually sufficient. This dimension decreases during pre-training and is lower in larger models. A lower dimension is also accompanied by higher validation accuracy and a smaller generalization gap. Finally, the authors derive a generalization bound that depends on d rather than on the full number of parameters. The work offers a theoretical explanation for the success of fine-tuning in low-data regimes and supports parameter-efficient fine-tuning methods.

Key points

  • RoBERTa on MRPC reaches 90% of the performance of full fine-tuning when only about 200 parameters are trained. According to Table 1 (the SAID method), d90 for RoBERTa-Large is 207 on MRPC and 774 on QQP, and for BERT-Base it is 1608 and 8030.
  • SAID consistently yields lower d90 estimates than the DID method, which does not take the model's division into layers into account.
  • For RoBERTa, a dense projection matrix at d = 1000 would take approximately 1.3–1.4 TB of memory. The authors therefore use Fastfood: a product of Hadamard matrices, random diagonal matrices and a permutation, computed in O(D log d).
  • Figure 3 compares models ranging from approximately 10⁸ to 10⁹+ parameters on MRPC: the larger the model, the smaller its d90. During RoBERTa pre-training, d90 gradually decreases.
  • The generalization bound is of order √(d/m), where m is the number of examples. It extends the compression bound from Arora et al. (2018). The empirical relationship with the generalization gap is verified on MRPC, QQP, Yelp, SST-2, MNLI and ANLI.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

195 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 191 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.568 · arXiv 2012.13255

@inproceedings{aghajanyan2021intrinsic,
  title = {{Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning}},
  author = {Aghajanyan, Armen and Gupta, Sonal and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.568},
  eprint = {2012.13255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.568}
}

References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03156. — [arXiv:2008.03156](https://arxiv.org/abs/2008.03156) find
  2. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. 2018. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296. — [arXiv:1802.05296](https://arxiv.org/abs/1802.05296) find
  3. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained bert networks. Advances in Neural Information Processing Systems, 33. find
  4. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019a. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. find
  5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019b. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. find
  6. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  8. Shrey Desai, Hongyuan Zhan, and Ahmed Aly. 2019. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12708. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  10. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  11. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620. find
  12. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  13. Geoffrey E Hinton and Richard Zemel. 1993. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6:3–10. find
  14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751. find
  15. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs. find
  16. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  17. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  19. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. find
  20. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561. find
  21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  25. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. find
  26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  27. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, pages arXiv–1910. find
  29. Quoc Le, Tamás Sarlós, and Alex Smola. 2013. Fastfood-approximating kernel expansions in loglin-ear time. In Proceedings of the international conference on machine learning, volume 85. find
  30. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763. find our card
  31. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs]. find our card