Elamur Research
☰
home / papers / 2021 / Parameter-Efficient Transfer Learning with Diff Pruning

Parameter-Efficient Transfer Learning with Diff Pruning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.378.pdf

Summary

Finetuning a large pretrained network separately for each task means storing a full copy of its weights per task. This is impractical when storage is limited, such as in on-device settings. Guo, Rush and Kim propose diff pruning. The pretrained weights stay frozen and shared, and each task learns only an additive difference vector that is pushed toward sparsity. Sparsity comes from a differentiable relaxation of the L0 penalty (Louizos et al., 2018), which applies a stretched Hard-Concrete mask to a dense weight vector. After training, the authors cut the vector to an exact budget by magnitude-pruning the diff values, then finetune again with the mask fixed. A structured variant groups parameters so that changes concentrate in local regions. Tasks are learned independently, so new ones can be added later without revisiting earlier tasks. On GLUE with BERT, diff pruning matches full finetuning while changing about 0.5% of parameters per task. This makes serving many tasks from one shared model much cheaper.

Key points

  • Each task's parameters are the frozen pretrained vector plus a sparse task-specific diff. Only the nonzero positions and their values need to be stored, so the base model's cost is spread across all tasks.
  • On the GLUE benchmark, diff pruning matches fully finetuned BERT baselines with roughly 0.5% new parameters per task. Adapters (Houlsby et al., 2019) need about 3.6%, and compression or pruning methods often keep 10–30% nonzero weights.
  • Table 6 reports per-task storage in a basic Python implementation (float32 weights, int32 positions):
    • full finetuning: 1297.0 MB
    • Adapters: 51.7 MB
    • diff pruning: 13.6 MB
  • An exact sparsity rate is set by projecting onto an L0-ball after training (keeping the top t% of diff magnitudes) and then finetuning further with the mask fixed. The structured variant gives additional gains, and the method sometimes acts as a regularizer that beats regular finetuning.
  • Code is available at github.com/dguo98/DiffPruning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 137 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.378

@inproceedings{guo2021parameter,
  title = {{Parameter-Efficient Transfer Learning with Diff Pruning}},
  author = {Guo, Demi and Rush, Alexander and Kim, Yoon},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.378},
  url = {https://doi.org/10.18653/v1/2021.acl-long.378}
}

References

67 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A Simple but Tough-to-Beat Baseline for Sentence Embeddings . In Proceedings of ICLR, 2017. find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Steven Cao, Victor Sanh, and Alexander M. Rush. Low-Complexity Probing via Finding Subnetworks. In Proceedings of NAACL, 2021. find
  4. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Deming Chen, Marianne Winslett, Hassan Sajjad, and Preslav Nakov. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. arXiv:2002.11985, 2020. find
  5. Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning. In Proceedings of Rep4NLP 2020 Workshop at ACL 2020, 2020. find
  6. Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spot-Tune: Transfer Learning Through Adaptive Fine-Tuning. In Proceedings of CVPR, 2019. find
  7. Song Han, Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In Proceedings of ICLR, 2016. find
  8. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of ACL, 2016. find our card
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, and Mona Attariyanand Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of ICML, 2019. find
  10. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL, 2018. find
  11. Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In Proceedings of ICLR, 2017. find our card
  12. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Proceedings of EMNLP (Findings), 2020. find
  13. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc… find
  14. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-Thought Vectors. In Proceedings of NeurIPS, 2015. find
  15. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In Proceedings of ICLR, 2020. find
  16. Quoc V. Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. In Proceedings of ICML, 2014. find our card
  17. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models. In Proceedings of ICLR, 2020. find
  18. Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems. 2017. find
  19. Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190, 2021. find
  20. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of ACL, 2019a. find our card
  21. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of ACL, 2019b. find our card
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692, 2019c. find
  23. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In Proceedings of NeurIPS, 2017. find
  24. Christos Louizos, Max Welling, Diederik P, and Kingma. Learning Sparse Neural Networks through L 0 Regularization. In Proceedings of ICLR, 2018. find
  25. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In Proceedings of ICLR, 2017. find our card
  26. Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights. In Proceedings of ECCV, 2018. find
  27. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Proceedings of NeurIPS. 2017. find our card
  28. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of EMNLP, 2017. find
  29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781, 2013. find
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of NAACL, 2018. find
  31. Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, and Kyunghyun Cho amd Iryna Gurevych. Adapter-Fusion: Non-Destructive Task Composition for Transfer Learning. arXiv:2005.00247, 2020a. find our card
  32. Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych Kyunghyun Cho. Adapter-Hub: A Framework for Adapting Transformers. arXiv:2007.07779, 2020b. — [arXiv:2007.07779](https://arxiv.org/abs/2007.07779) find
  33. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. arXiv:2005.00052, 2020c. find our card
  34. Guanghui Qin and Jason Eisner. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of NAACL, 2021. find our card
  35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. find
  36. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. find
  37. Evani Radiya-Dixit and Xin Wang. How fine can fine-tuning be? Learning efficient language models. In Proceedings of AISTATS, 2020. find
  38. Colin Raffel, Noam Shazeer, Katherine Lee Adam Roberts, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21, 2020. find
  39. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054, 2019. find
  40. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, 2016. find our card
  41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053, 2019. find
  42. Asa Cooper Stickland and Iain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of ICML, 2019. find
  43. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. In Proceedings of ICLR, 2018. find
  44. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of EMNLP, 2019. find our card
  45. S. Rebuffi, A. Vedaldi, and H. Bilen. Efficient Parametrization of Multi-domain Deep Neural Networks. In Proceedings of CVPR, 2018. find
  46. Ximeng Sun, Rameswar Panda, and Rogerio Feris. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning. In Proceedings of NeurIPS, 2020a. find
  47. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP, 2019. find
  48. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of ACL, July 2020b. find our card
  49. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv:1606.04671, 2016. find
  50. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor Man’s BERT: Smaller and Faster Transformer Models. arXiv:2004.03844, 2020. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) find
  51. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019. find
  52. Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning. arXiv:2005.07683, 2020. — [arXiv:2005.07683](https://arxiv.org/abs/2005.07683) find
  53. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACL, 2019. find our card
  54. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. arXiv:1908.08962, 2019. find our card
  55. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR, 2019a. find our card
  56. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured Pruning of Large Language Models. arXiv:1910.04732, 2019b. find
  57. Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. In Proceedings of CoNLL, August 2017. find
  58. Timo Schick and Hinrich Schutze. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv:2009.07118, 2020. find our card
  59. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards Universal Paraphrastic Sentence Embeddings. In Proceedings of ICLR, 2016. find our card
  60. Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & Compress: A scalable framework for continual learning. In Proceedings of ICML, 2018. find
  61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  62. Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In Proceedings of NeurIPS. 2017. find
  63. John M. Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity Analysis of Contextual Word Representation Models. In Proceedings of ACL, 2020. find
  64. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of NeurIPS, 2019. find our card
  65. Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks. In Proceedings of ECCV, 2020a. find
  66. Minghua Zhang, Yunfang Wu, Weikang Li, and Wei Li. Learning universal sentence representations with mean-max attention autoencoder. In Proceedings of EMNLP, 2018. find
  67. Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An Unsupervised Sentence Embedding Method byMutual Information Maximization. In Proceedings of EMNLP, 2020b. find