Elamur Research⌕
☰
home / papers / 2022 / Structured Pruning Learns Compact and Accurate Models

Structured Pruning Learns Compact and Accurate Models

Summary

Compressing large pre-trained language models usually involves a trade-off. Distillation produces fast compact models but needs costly general training on large unlabeled corpora. Structured pruning is cheaper but has seldom delivered more than 2–3× speedups. The authors propose CoFi (Coarse- and Fine-grained Pruning), a task-specific structured pruning method for Transformers. It starts from a fine-tuned BERT model and learns several mask types at once. Coarse masks remove whole MHA or FFN layers, while fine masks remove individual heads, FFN intermediate dimensions and hidden dimensions. Each parameter is therefore governed by masks at multiple granularities. CoFi also adds a layerwise distillation objective that learns, during training, how layers of the shrinking student map onto the unpruned teacher. On GLUE and SQuAD v1.1, CoFi matches or beats strong distillation and pruning baselines in both accuracy and latency. It exceeds 10× speedup at roughly 95% sparsity and uses no extra unlabeled data. This makes structured pruning a practical, low-compute alternative to distillation for extreme compression.

Key points

  • CoFi learns five mask variables jointly: z_FFN, z_int, z_MHA, z_head and z_hidn. In ablations, dropping the layer-level masks greatly reduces speedup at high sparsity, and dropping the hidden-dimension masks noticeably hurts accuracy.
  • On MNLI, measured on an NVIDIA V100 against BERT base (84.8, 85M params), CoFi reaches 84.9 at 2.7× speedup with 26M params. It reaches 80.6 at 12.1× speedup with 4.4M params, which beats TinyBERT 4 at 11.4× (78.8).
  • Training cost is low. CoFi needs at most about 20 hours on one GPU per GLUE task, and under 3 hours for smaller datasets. For comparison, TinyBERT's general distillation alone takes about 3.5 days on 4 GPUs over 2,500M tokens.
  • The pruned architectures differ by dataset. They tend to keep more MHA than FFN layers and usually preserve the first two MHA layers, while middle layers are often removed.
  • Code and models are released at github.com/princeton-nlp/CoFiPruning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.107

@inproceedings{xia2022structured,
  title = {{Structured Pruning Learns Compact and Accurate Models}},
  author = {Xia, Mengzhou and Zhong, Zexuan and Chen, Danqi},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.107},
  url = {https://doi.org/10.18653/v1/2022.acl-long.107}
}

References

55 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… find
  2. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020a. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 15834– 15846. find
  3. Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. 2020b. Earlybert: Efficient bert training via early-bird lottery tickets. arXiv preprint arXiv:2101.00063. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171–4186. find
  5. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  6. Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing Transformer depth on demand with structured dropout. In International Conference on Learning Representations (ICLR). find our card
  7. Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In International Conference on Learning Representations (ICLR). find
  8. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR). find
  9. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), pages 3259–3269. find
  10. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. 2021. Compressing large-scale Transformer-based models: A case study on BERT. Transactions of the Association of Computational Linguistics (TACL), 9:1061–1080. find
  11. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. DynaBERT: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. find
  12. Shaoyi Huang, Dongkuan Xu, Ian EH Yen, Sung-en Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Hang Liu, and Caiwen Ding. 2021. Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm. arXiv e-prints, pages arXiv– 2110. find
  13. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of Empirical Methods in Natural Language Processing (EMNLP), pages 4163– 4174. find
  14. François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. 2021. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838. find
  15. Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2021. Differentiable subset pruning of Transformer heads. Transactions of the Association of Computational Linguistics (TACL). find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  17. Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019b. Metapruning: Meta learning for automatic neural network channel pruning. In International Conference on Computer Vision (ICCV), pages 3296–3305. find
  18. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision (ICCV), pages 2736–2744. find
  19. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019c. Rethinking the value of network pruning. In International Conference on Learning Representations (ICLR). find
  20. Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. 2020. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  21. C Louizos, M Welling, and DP Kingma. 2018. Learning sparse neural networks through l0 regularization. In International Conference on Learning Representations (ICLR). find
  22. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In International Conference on Computer Vision (ICCV), pages 1389–1397. find
  23. Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017. Thinet: A filter level pruning method for deep neural network compression. In International Conference on Computer Vision (ICCV), pages 5058–5066. find
  24. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  25. JS McCarley, Rishav Chakravarti, and Avirup Sil. 2019. Structured pruning of a BERT-based question answering model. arXiv preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) find
  26. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems (NeurIPS). find our card
  27. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11264–11272. find
  28. Matan Ben Noach and Yoav Goldberg. 2020. Compressing pre-trained language models by matrix decomposition. In Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884–889. find
  29. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In Empirical Methods in Natural Language Processing (EMNLP), pages 3208–3229. find
  30. Ofir Press, Noah A Smith, and Omer Levy. 2020. Improving transformer models by reordering their sub-layers. In Association for Computational Linguistics (ACL), pages 2996–3005. find
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. The Journal of Machine Learning Research (JMLR), 21(140). find
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. find our card
  33. Alex Renda, Jonathan Frankle, and Michael Carbin. 2020. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations (ICLR). find
  34. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. Poor man’s BERT: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) find
  35. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  36. Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems (NeurIPS), 33. find
  37. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-BERT: Hessian based ultra low precision quantization of BERT. In Conference on Artificial Intelligence (AAAI), pages 8815–8821. find
  38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP). find
  39. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Empirical Methods in Natural Language Processing (EMNLP), pages 4314–4323. find our card
  40. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic bert for resource-limited devices. In Association for Computational Linguistics (ACL), pages 2158–2170. find our card
  41. Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19. find our card
  42. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962. find our card
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems (NIPS), 30:5998–6008. find
  44. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Association for Computational Linguistics (ACL), pages 5797–5808. find our card
  45. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR). find our card
  46. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020a. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained Transformers. In Advances in Neural Information Processing Systems (NeurIPS). find
  47. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020b. Structured pruning of large language models. In Empirical Methods in Natural Language Processing (EMNLP), pages 6151–6162. find our card
  48. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. In Transactions of the Association of Computational Linguistics (TACL), volume 7, pages 625–641. find
  49. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  50. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Association for Computational Linguistics (ACL), pages 2246–2251. find our card
  51. Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. 2021. Rethinking network pruning–under the pre-train and fine-tune paradigm. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 2376–2382. find
  52. Zhewei Yao, Linjian Ma, Sheng Shen, Kurt Keutzer, and Michael W Mahoney. 2021. MLPruning: A multilevel structured pruning framework for transformer-based models. arXiv preprint arXiv:2105.14636. — [arXiv:2105.14636](https://arxiv.org/abs/2105.14636) find
  53. Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2021. AutoTinyBERT: Automatic hyper-parameter optimization for efficient pre-trained language models. In Association for Computational Linguistics (ACL), pages 5146–5157. find
  54. Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. 2021. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.05754. — [arXiv:2111.05754](https://arxiv.org/abs/2111.05754) find
  55. Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. 2019. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/