Elamur Research⌕
☰
home / papers / 2016 / Learning Structured Sparsity in Deep Neural Networks

Learning Structured Sparsity in Deep Neural Networks

Paper · 2016 · Wei Wen, Chunpeng Wu, Yandan Wang et al.

Summary

Sparsifying a deep network weight by weight shrinks its parameter count, but the zeros end up scattered across memory. Standard CPU and GPU libraries then gain little speed and can even run slower. The authors propose Structured Sparsity Learning (SSL) for convolutional layers. During training, a group Lasso penalty pushes whole blocks of weights to zero at once: entire filters, entire channels, spatial positions inside filters (giving non-cuboid filter shapes), or complete layers. Removing a whole layer is safe in residual-style networks because shortcut connections still carry the signal. The surviving weights stay in dense, contiguous blocks, so ordinary GEMM routines can exploit the savings.

On AlexNet trained on ILSVRC 2012, SSL speeds up convolutional layers about 5.1× on CPU and 3.1× on GPU with off-the-shelf libraries, roughly double what unstructured ℓ1 sparsity achieves. Used as a regularizer, it can also improve accuracy. The work matters because it brings compression in line with how hardware actually executes models.

Key points

  • Structures penalized: filter-wise, channel-wise, shape-wise and depth-wise groups, plus finer variants (2D-filter-wise, and row/column sparsity of the GEMM-lowered weight matrix). Code is released as a Caffe fork (github.com/wenwei202/caffe, branch scnn).
  • AlexNet speedups: SSL reaches on average 5.1× layer-wise speedup on CPU (Intel Xeon, MKL) and 3.1× on GPU (GTX TITAN Black, cuBLAS). Unstructured ℓ1 sparsity with CSR/cuSPARSE gives only 3.0× and 0.9× at similar error.
  • Unstructured sparsity barely helps: even above 95% sparsity, unstructured ℓ1 sparsification yields small or negative GPU speedups, because memory access becomes irregular.
  • Depth regularization on CIFAR-10: a 20-layer ResNet was cut to 18 layers while accuracy rose from 91.25% to 92.60%. That is slightly better than the original 32-layer ResNet.
  • AlexNet accuracy and FLOPs: 2D-filter-wise regularization lowers FLOPs by 30–40% with no accuracy loss. Alternatively, if all parameters are kept, it reduces top-1 error by about 1% (to 41.69%). Shape-wise regularization gives similar results (41.83%).

Interesting ideas

  • I1. Match the penalty to hardware units: choose sparsity groups that line up with how computation is laid out, such as rows and columns of a matrix multiply. The same principle could make pruned NLP models, for example attention heads or FFN blocks, run faster on stock hardware instead of only looking smaller on paper.
  • I2. Penalize coupled structures together: a filter that becomes zero makes the matching input channel of the next layer useless. Regularizing filters and channels jointly exploits this dependency, and a similar trick fits any architecture whose components are chained.
  • I3. Remove layers by regularization, not manual search: with shortcut connections, a whole layer can be zeroed without breaking the forward pass. Depth then becomes a learned property, which beats repeated manual architecture trials.
  • I4. Evaluate with wall-clock time: the authors measure actual speedups with standard libraries on several CPUs and GPUs rather than only counting zeros. This protocol exposes compression methods that look efficient but deliver no real gains.
  • I5. Combine with low-rank approximation: weights regularized by SSL lie in a lower-dimensional space, as their PCA analysis shows. So SSL can be stacked with low-rank approximation for further compression, a useful pairing whenever several compression steps are chained.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States65 works · 23,401 citations🎓Kobe UniversityKobe · Japan2 works · 530 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

Cite

Cite as Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen and Hai Li. 2016. Learning Structured Sparsity in Deep Neural Networks. arXiv:1608.03665.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1608.03665 · arXiv 1608.03665

@misc{wen2016learning,
  title = {{Learning Structured Sparsity in Deep Neural Networks}},
  author = {Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen and Hai Li},
  year = {2016},
  doi = {10.48550/arxiv.1608.03665},
  eprint = {1608.03665},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.03665}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105. 2012.
  2. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  3. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  4. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2015. find
  5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
  6. Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. find
  7. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135–1143. 2015. find
  8. Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015. find
  9. Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pages 2148–2156. 2013. find
  10. Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pages 1269–1277. 2014. find
  11. Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014. find
  12. Yani Ioannou, Duncan P. Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv:1511.06744, 2015. — [arXiv:1511.06744](https://arxiv.org/abs/1511.06744) find
  13. Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067, 2015. — [arXiv:1511.06067](https://arxiv.org/abs/1511.06067) find
  14. Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 68(1):49–67, 2006. find
  15. Seyoung Kim and Eric P Xing. Tree-guided group lasso for multi-task regression with structured sparsity. In Proceedings of the 27th International Conference on Machine Learning, 2010. find
  16. Jiashi Feng and Trevor Darrell. Learning the structure of deep convolutional networks. In The IEEE International Conference on Computer Vision (ICCV), 2015. find
  17. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.
  18. Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
  19. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  20. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  21. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2016-learning-structured-sparsity-deep-neural/