Elamur Research⌕
☰
home / papers / 2021 / Block Pruning For Faster Transformers

Block Pruning For Faster Transformers

Paper · 2021 · François Lagunas, Ella Charlaix, Victor Sanh et al.

Summary

Large pre-trained transformers are accurate but expensive to store and run. Unstructured pruning shrinks them a lot but rarely speeds up inference on ordinary hardware. Distilled models such as DistilBERT, TinyBERT or MobileBERT are fast but stay relatively large. The Hugging Face authors close this gap by extending movement pruning, which learns importance scores during fine-tuning. In their version, a single score and mask is shared by each fixed-size block of a weight matrix, such as 32×32 squares or whole FFN rows and columns. Attention and feed-forward weights get separate regularization strengths, and a teacher model supplies distillation. They test BERT on SQuAD v1.1, SQuAD v2, MNLI, QQP and SST-2, and BART on CNN/DailyMail. Unexpectedly, training with small square blocks ends up deleting entire attention heads. The remaining matrices stay dense, which produces real speedups. On SQuAD v1.1, a pruned BERT-base runs 2.4x faster and is 74% smaller for about a 1% F1 loss. The method works at fine-tuning time and needs no architecture search or data augmentation.

Key points

  • Efficiency results:
    • SQuAD v1.1 (BERT-base): 2.4x speedup with ~1% F1 drop.
    • QQP: 2.3x speedup with ~1% F1 drop.
    • CNN/DailyMail (BART): 1.39x speedup and 3.5x fewer decoder weights, losing about 2 ROUGE points on average.
  • BERT-large results:
    • A model with 18% of its weights left reaches F1 90.2 on SQuAD v1.1 (vs 93.2 unpruned) at 3.2x speedup, faster than BERT-base.
    • On SQuAD v2, a 17%-dense Hybrid model gets F1 82.6, beating McCarley's (2019) structured pruning (25% dense, F1 81.5).
  • Block size: All block sizes from 4 to 32 can recover BERT-base F1 on SQuAD v1.1. Larger blocks give more speed, with gains leveling off around size 16; 32×32 blocks allow 1.75x speedup for a 2% F1 loss.
  • Quantization and distillation: Adding 8-bit quantization yields 13.3x compression of BERT-base with F1 86.3 on SQuAD v1.1. Distillation consistently helps but is not essential, and it matters most on small datasets like SST-2.
  • Code: Released as huggingface/nn_pruning on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🏢Huawei TechnologiesShenzhen · China29 works · 9,991 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 77 works

Cite

Cite as François Lagunas, Ella Charlaix, Victor Sanh and Alexander Rush. 2021. Block Pruning For Faster Transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.829

@inproceedings{lagunas2021block,
  title = {{Block Pruning For Faster Transformers}},
  author = {François Lagunas, Ella Charlaix, Victor Sanh and Alexander Rush},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.829},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.829}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693–1701. find
  2. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
  3. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling BERT for natural language understanding. CoRR, abs/1909.10351. find
  4. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomás Mikolov. 2016. Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651.
  5. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432. find
  6. Young Jin Kim and Hany Hassan Awadalla. 2020. Fast-formers: Highly efficient transformer models for natural language understanding. find
  7. Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. Quora question pairs. University of Waterloo. find
  8. Yann LeCun, John S. Denker, and Sara A. Solla. 1989. Optimal brain damage. In Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pages 598–605. Morgan Kaufmann. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  11. Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Training pruned neural networks. CoRR, abs/1803.03635. find
  12. Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. CoRR, abs/2002.08307.
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  14. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. 2020. Train large, then compress: Rethinking model size for efficient training and inference of transformers. CoRR, abs/2002.11794. — [arXiv:2002.11794](https://arxiv.org/abs/2002.11794) find
  15. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through l_0 regularization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. find
  16. Demi Guo, Alexander M. Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. CoRR, abs/2012.07463.
  17. Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, and Jing Bai. 2020. Ladabert: Lightweight adaptation of BERT through hybrid model compression. CoRR, abs/2004.04124. find
  18. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. CoRR, abs/1506.02626. find
  19. J. S. McCarley. 2019. Structured pruning of a bert-based question answering model. CoRR, abs/1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) find
  20. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? CoRR, abs/1905.10650.
  21. Kenton Murray and David Chiang. 2015. Auto-sizing neural networks: With applications to ngram language models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 908–916. The Association for Computatio… find
  22. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. Associa…
  23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The A…
  24. Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. 2019. What’s hidden in a randomly weighted neural network? CoRR, abs/1911.13299. — [arXiv:1911.13299](https://arxiv.org/abs/1911.13299) find
  25. Corby Rosset. 2020. Turing-nlg: A 17-billion-parameter language model by microsoft. Microsoft Research Blog, 2:13. find
  26. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. Poor man’s BERT: smaller and faster transformer models. CoRR, abs/2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) find
  27. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  28. Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. CoRR, abs/2005.07683. — [arXiv:2005.07683](https://arxiv.org/abs/2005.07683) find
  29. Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of neural machine translation models via pruning. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 291–301. ACL. find
  30. SIGDAT, a Special Interest Group of the ACL, pages 1631–1642. ACL. find
  31. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong…
  32. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic BERT for resource-limited devices. CoRR, abs/2004.02984.
  33. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.12136. find
  34. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962.
  35. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. CoRR, abs/1905.09418.
  36. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019. Structured pruning of large language models. CoRR, abs/1910.04732.
  37. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018… find
  38. Sam Shleifer and Alexander M. Rush. 2020. Pre-trained summarization distillation. CoRR, abs/2010.13002. — [arXiv:2010.13002](https://arxiv.org/abs/2010.13002) find
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 20… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lagunas-2021-block-pruning-faster-transformers/