Elamur Research⌕
☰
home / papers / 2021 / Accelerating Sparse Deep Neural Networks

Accelerating Sparse Deep Neural Networks

Paper · 2021 · Asit Mishra, Jorge Albericio Latorre, Jeff Pool et al.

Summary

Pruning can shrink neural networks, but it rarely delivers both full accuracy and real speed. Unstructured sparsity keeps accuracy yet maps poorly onto GPU matrix units. Coarse structured sparsity runs fast but loses accuracy once sparsity gets high. The NVIDIA authors propose a fine-grained structured pattern called 2:4, where every four consecutive values contain at least two zeros. That gives 50% sparsity. They pair it with a compact storage format and with Sparse Tensor Cores in the Ampere GPU architecture, which double matrix-multiply math throughput when one operand is a compressed 2:4 tensor.

To produce compatible models, they use a simple recipe:

  • train a dense network,
  • prune it once,
  • repeat the original training run with the sparsity mask frozen and the same hyper-parameters.

Across vision, translation and language-modeling networks, the sparse models match their dense counterparts, including under INT8 quantization. The work matters because it offers a single sparsity scheme that is both accurate and genuinely accelerated in commodity hardware, without per-model tuning.

Key points

  • Storage format. A 2:4 sparse matrix of size R × C is stored as R × C/2 nonzero values plus a 2-bit index per value. The metadata overhead is 12.5% for 16-bit weights and 25% for 8-bit weights. Total storage drops by about 44% (FP16) and about 38% (INT8) compared with dense.
  • Hardware. Sparse Tensor Cores give 2× math throughput for GEMM-like operations (convolutions, matrix multiplies) whose first argument is 2:4 sparse.
  • Translation (BLEU, WMT'14 En-De). GNMT goes from 24.6 dense to 24.9 sparse FP16 and 24.9 sparse INT8. The FairSeq Transformer goes from 28.2 to 28.5 (FP16) and 28.3 (INT8).
  • Language modeling. Transformer-XL on enwik8 stays at 1.06 BPC. BERT LARGE on SQuAD v1.1 keeps 91.9 F1 in FP16, and scores 90.8 sparse vs 90.9 dense in INT8. For pre-trained models, pre-training is rerun after pruning, then the model is fine-tuned. Attention products are left unpruned because they contain no weights.
  • Tooling and limits. Masks come from the PyTorch ASP library, which offers greedy or exhaustive search to minimize lost weight magnitude. The workflow's main cost is that it doubles training time.

Interesting ideas

  • I1. Hardware-friendly mid-level sparsity. Choosing a moderate, uniform sparsity level (50% in every small group) instead of extreme unstructured sparsity keeps accuracy recoverable and memory access predictable. This trade-off could guide compression design for any model meant for real deployment.
  • I2. Retrain without tuning. Pruning once and then replaying the original training recipe with a fixed mask avoids hyper-parameter searches. It is a robust baseline that can be reused across many architectures and tasks, including Ukrainian NLP models.
  • I3. Fixed-size groups with tiny metadata. Tiny fixed-size groups mean the position of every stored value can be computed directly, with no data-dependent indirection. This idea is useful for any compact tensor or embedding storage.
  • I4. Prune, then redo pre-training. For pre-trained language models, pruning and then repeating pre-training before task fine-tuning was what recovered dense accuracy. This is a practical recipe for compressing BERT-like models for downstream tasks.
  • I5. Test sparsity together with INT8. Checking that sparsity and INT8 quantization compose without extra loss is a good evaluation habit, because deployment usually combines several compression methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NvidiaSanta Clara · United States37 works · 12,656 citations🏢Nvidia (United Kingdom)Reading · United Kingdom27 works · 11,141 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu and Paulius Micikevicius. 2021. Accelerating Sparse Deep Neural Networks. arXiv 2021 (NVIDIA).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2104.08378

@misc{mishra2021accelerating,
  title = {{Accelerating Sparse Deep Neural Networks}},
  author = {Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu and Paulius Micikevicius},
  year = {2021},
  eprint = {2104.08378},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.08378}
}

References

65 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00409, 2017.
  2. Joel Hestness, Newsha Ardalani, and Greg Diamos. Beyond human-level accuracy: Computational challenges in deep learning. CoRR, abs/1909.01736, 2019. — [arXiv:1909.01736](https://arxiv.org/abs/1909.01736) find
  3. Md. Mostofa Ali Patwary, Milind Chabbi, Heewoo Jun, Jiaji Huang, Gregory Frederick Diamos, and Kenneth Church. Language modeling at scale. CoRR, abs/1810.10045, 2018. find
  4. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Mesh-tensorflow: Deep learning for supercomputers. CoRR, abs/1811.02084, 2018.
  5. Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In EUROSPEECH, 1997. find
  6. Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, and William J. Dally. DSD: regularizing deep neural networks with dense-sparse-dense training flow. In ICLR, 2017. find
  7. Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 1135–1143, 2015. find
  8. Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Exploring the granularity of sparsity in convolutional neural networks. In CVPR Workshops, pages 13–20, 2017. find
  9. Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In ICLR Workshops, 2018. find
  10. Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. CoRR, abs/1707.06168, 2017. — [arXiv:1707.06168](https://arxiv.org/abs/1707.06168) find
  11. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. CoRR, abs/1611.06440, 2016. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) find
  12. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. CoRR, abs/1810.05270, 2018. find
  13. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR, abs/1803.03635, 2018. find
  14. NVIDIA A100 tensor core GPU architecture. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper. pdf, 2020. ↗
  15. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, 2019. find
  16. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: single-shot network pruning based on connection sensitivity. In ICLR, 2019. find
  17. Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. Balanced sparsity for efficient DNN inference on GPU. CoRR, abs/1811.00206, 2018. find
  18. Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured pruning for efficient convnets via incremental regularization. CoRR, abs/1811.08390, 2018. — [arXiv:1811.08390](https://arxiv.org/abs/1811.08390) find
  19. Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. CoRR, abs/1512.08571, 2015. — [arXiv:1512.08571](https://arxiv.org/abs/1512.08571) find
  20. Mingbao Lin, Rongrong Ji, Shaojie Li, Qixiang Ye, Yonghong Tian, Jianzhuang Liu, and Qi Tian. Filter sketch for network pruning. CoRR, abs/2001.08514, 2020. — [arXiv:2001.08514](https://arxiv.org/abs/2001.08514) find
  21. Noah Gamboa, Kais Kudrolli, Anand Dhoot, and Ardavan Pedram. Campfire: Compressible, regularization-free, structured sparse training for hardware accelerators. CoRR, abs/2001.03253, 2020. — [arXiv:2001.03253](https://arxiv.org/abs/2001.03253) find
  22. Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In MICRO, 2019. find
  23. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. CoRR, abs/1608.03665, 2016.
  24. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The lottery ticket hypothesis at scale. CoRR, abs/1903.01611, 2019. — [arXiv:1903.01611](https://arxiv.org/abs/1903.01611) find
  25. Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In ICLR, 2020. find
  26. Sharan Narang, Gregory F. Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. CoRR, abs/1704.05119, 2017. find
  27. Sharan Narang, Eric Undersander, and Gregory F. Diamos. Block-sparse recurrent neural networks. CoRR, abs/1711.02782, 2017. — [arXiv:1711.02782](https://arxiv.org/abs/1711.02782) find
  28. Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. CoRR, abs/1911.09723, 2019. — [arXiv:1911.09723](https://arxiv.org/abs/1911.09723) find
  29. Mi Sun Park, Xiaofan Xu, and Cormac Brick. Squantizer: Simultaneous learning for both sparse and low-precision neural networks. CoRR, abs/1812.08301, 2018. — [arXiv:1812.08301](https://arxiv.org/abs/1812.08301) find
  30. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. CoRR, abs/1802.08435, 2018. find
  31. Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. CoRR, abs/1911.11134, 2019. — [arXiv:1911.11134](https://arxiv.org/abs/1911.11134) find
  32. Scott Gray, Alec Radford, and Diederik P. Kingma. GPU Kernels for Block-Sparse Weights, Accessed April 13, 2021. https://openai.com/blog/block-sparse-gpu-kernels/. ↗
  33. H. Kang. Accelerator-aware pruning for convolutional neural networks. TCSVT, 30(7):2093–2103, 2020. find
  34. Sparse matrix storage formats. https://www.gnu.org/software/gsl/doc/html/spmatrix.html. ↗
  35. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  36. NVIDIA’s Automatic SParsity (ASP) library. https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity, 2020. ↗
  37. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
  38. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. find
  39. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211–252, 2015.
  40. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014.
  41. Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. CoRR, abs/2004.09602, 2020. find
  42. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
  43. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  44. Sohil Shah, Pallabi Ghosh, Larry S. Davis, and Tom Goldstein. Stacked u-nets: A no-frills approach to natural image segmentation. CoRR, abs/1804.10343, 2018. — [arXiv:1804.10343](https://arxiv.org/abs/1804.10343) find
  45. Fisher Yu, Vladlen Koltun, and Thomas A. Funkhouser. Dilated residual networks. CoRR, abs/1705.09914, 2017. find
  46. Dhruv Kumar Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. find
  47. I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. CoRR, abs/1905.00546, 2019. — [arXiv:1905.00546](https://arxiv.org/abs/1905.00546) find
  48. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. find
  49. Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \<1mb model size. CoRR, abs/1602.07360, 2016. find
  50. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-aware neural architecture search for mobile. In CVPR, 2019. find
  51. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. find
  52. Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
  53. Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2, 2019. ↗
  54. Nvidia deep learning examples. https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD. ↗
  55. Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In CVPR, 2019. find
  56. Chong Yu and Jeff Pool. Self-supervised generative adversarial compression. In NeurIPS, 2020. find
  57. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. find
  58. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  59. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT: Demonstrations, 2019.
  60. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860, 2019. find
  61. The Hutter Prize Wikipedia dataset, Accessed April 13, 2020. http://prize.hutter1.net/. ↗
  62. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250, 2016.
  63. Megatron-LM repo, Accessed April 13, 2020. https://github.com/NVIDIA/Megatron-LM. ↗
  64. Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n:m transposable masks. CoRR, abs/2102.08124, 2021. — [arXiv:2102.08124](https://arxiv.org/abs/2102.08124) find
  65. Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M fine-grained structured sparse neural networks from scratch. In ICLR, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2021-accelerating-sparse-deep-neural-networks/