Elamur Research⌕
☰
home / papers / 2017 / Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Paper · 2017 · Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang et al.

Summary

DNNs reach top accuracy in vision, speech, robotics and other AI tasks, but they demand heavy computation. This makes them expensive to deploy, especially beyond general-purpose GPUs. Sze, Chen, Yang and Emer (MIT and Nvidia) wrote a combined tutorial and survey on running DNNs faster and with less energy without hurting accuracy or raising hardware cost. It is not a single experiment. The article first places deep learning within AI, machine learning and brain-inspired computing, then explains DNN building blocks, popular models, and tools for research and development. Its core sorts efficiency techniques into three groups:

  • hardware-only optimizations that keep outputs bit-wise identical;
  • near-data processing with mixed-signal circuits and new memory technologies, aimed at costly data movement;
  • joint algorithm–hardware co-design that trades a small amount of accuracy for speed and energy.

It ends with the metrics that fair comparisons of DNN hardware should use. The article matters as a structured entry point for both practitioners and researchers. It also offers a shared vocabulary for judging the many accelerator proposals from academia and industry.

Key points

  • Under the paper's definition, a network counts as deep when it has more than three layers, meaning more than one hidden layer. Typical depth runs from about five to over a thousand layers.
  • The paper separates two phases. Training finds the weights by reducing a loss, computed with backpropagation. Inference runs the network with those fixed weights. The paper uses image classification as its running example.
  • The survey covers co-design methods such as:
    • reduced precision (BinaryConnect, BinaryNet, XNOR-Net, ternary and logarithmic quantization);
    • pruning (Optimal Brain Damage, energy-aware pruning);
    • compact architectures (SqueezeNet, MobileNets, depthwise separable convolutions);
    • low-rank decomposition;
    • knowledge distillation.
  • Sparse and compressed accelerators discussed include EIE, SCNN, Cnvlutin and Minerva. Spiking hardware is represented by IBM TrueNorth.
  • The authors provide public resources: a DNN energy estimation tool and a benchmarking page for DNN processors, both on eyeriss.mit.edu.

Interesting ideas

  • I1. Split optimizations by whether they change outputs. The paper keeps techniques that give bit-wise identical results apart from those that trade accuracy for efficiency. Any efficiency report, for example on NLP model compression, gets clearer when lossless and lossy gains are kept separate.
  • I2. Make the cost you care about explicit. Energy-aware pruning removes weights according to estimated energy, not parameter count, so it optimizes the real deployment cost instead of a proxy. The same principle applies to latency, memory, or cloud cost when shrinking models.
  • I3. Treat data movement as the main bottleneck. The authors argue that moving data, more than arithmetic, dominates energy and throughput. Profiling memory traffic first, not FLOPs, is useful for any large data-processing pipeline.
  • I4. Report a full set of metrics. The paper argues that new designs should be judged on several considerations together, such as accuracy, throughput, energy and hardware cost, never a single headline number. This guards against cherry-picked comparisons in any benchmark the family publishes.
  • I5. Ship tools alongside the survey. Releasing an energy estimator and a benchmarking page with the paper lowers the entry barrier for newcomers. This is worth copying for our own resources, such as corpora or treebanks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang and Joel Emer. 2017. Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proceedings of the IEEE 2017.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1703.09039

@inproceedings{sze2017efficient,
  title = {{Efficient Processing of Deep Neural Networks: A Tutorial and Survey}},
  author = {Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang and Joel Emer},
  booktitle = {Proceedings of the IEEE 2017},
  year = {2017},
  eprint = {1703.09039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.09039}
}

References

162 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015. find
  2. L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, J. Williams et al., “Recent advances in deep learning for speech research at Microsoft,” in ICASSP, 2013. find
  3. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in NIPS, 2012.
  4. C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in ICCV, 2015. find
  5. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, 2017. find
  6. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go… find
  7. F.-F. Li, A. Karpathy, and J. Johnson, “Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition,” http://cs231n.stanford.edu/. ↗
  8. P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada, F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura et al., “A million spiking-neuron integrated circuit with a scalable communication network and interface,” Science, vol. 345, no. 6197, pp. 668–673, 2014. find
  9. S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, D. J. Berg, J. L. McKinstry, T. Melano, D. R. Barch et al., “Convolutional networks for fast, energy-efficient neuromorphic computing,” Proceedings of the National Academy of Sciences, 2016. find
  10. M. Mathieu, M. Henaff, and Y. LeCun, “Fast training of convolutional networks through FFTs,” in ICLR, 2014. find
  11. Y. LeCun, L. D. Jackel, B. Boser, J. S. Denker, H. P. Graf, I. Guyon, D. Henderson, R. E. Howard, and W. Hubbard, “Handwritten digit recognition: applications of neural network chips and automatic learning,” IEEE Commun. Mag., vol. 27, no. 11, pp. 41–46, Nov 1989. find
  12. B. Widrow and M. E. Hoff, “Adaptive switching circuits,” in 1960 IRE WESCON Convention Record, 1960. find
  13. B. Widrow, “Thinking about thinking: the discovery of the LMS algorithm,” IEEE Signal Process. Mag., 2005. find
  14. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp. 211–252, 2015.
  15. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in CVPR, 2016.
  16. “Complete Visual Networking Index (VNI) Forecast,” Cisco, June 2016. find
  17. J. Woodhouse, “Big, big, big data: higher and higher resolution video surveillance,” technology.ihs.com, January 2016. find
  18. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation,” in CVPR, 2014.
  19. J. Long, E. Shelhamer, and T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” in CVPR, 2015. find
  20. K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in NIPS, 2014. find
  21. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82–97, 2012. find
  22. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research, vol. 12, no. Aug, pp. 2493–2537, 2011.
  23. A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” CoRR abs/1609.03499, 2016.
  24. H. Y. Xiong, B. Alipanahi, L. J. Lee, H. Bretschneider, D. Merico, R. K. Yuen, Y. Hua, S. Gueroussov, H. S. Najafabadi, T. R. Hughes et al., “The human splicing code reveals new insights into the genetic determinants of disease,” Science, vol. 347, no. 6218, p. 1254806, 2015. find
  25. J. Zhou and O. G. Troyanskaya, “Predicting effects of noncoding variants with deep learning-based sequence model,” Nature methods, vol. 12, no. 10, pp. 931–934, 2015. find
  26. B. Alipanahi, A. Delong, M. T. Weirauch, and B. J. Frey, “Predicting the sequence specificities of dna-and rna-binding proteins by deep learning,” Nature biotechnology, vol. 33, no. 8, pp. 831–838, 2015. find
  27. H. Zeng, M. D. Edwards, G. Liu, and D. K. Gifford, “Convolutional neural network architectures for predicting dna–protein binding,” Bioinformatics, vol. 32, no. 12, pp. i121–i127, 2016. find
  28. M. Jermyn, J. Desroches, J. Mercier, M.-A. Tremblay, K. St-Arnaud, M.-C. Guiot, K. Petrecca, and F. Leblond, “Neural networks improve brain cancer detection with raman spectroscopy in the presence of operating room light artifacts,” Journal of Biomedical Optics, vol. 21, no. 9, pp. 094 002–094 002,… find
  29. D. Wang, A. Khosla, R. Gargeya, H. Irshad, and A. H. Beck, “Deep learning for identifying metastatic breast cancer,” arXiv preprint arXiv:1606.05718, 2016. — [arXiv:1606.05718](https://arxiv.org/abs/1606.05718) find
  30. L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of artificial intelligence research, vol. 4, pp. 237–285, 1996. find
  31. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” in NIPS Deep Learning Workshop, 2013. find
  32. S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 39, pp. 1–40, 2016. find
  33. M. Pfeiffer, M. Schaeuble, J. Nieto, R. Siegwart, and C. Cadena, “From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robots,” in ICRA, 2017. find
  34. S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive mapping and planning for visual navigation,” in CVPR, 2017. find
  35. T. Zhang, G. Kahn, S. Levine, and P. Abbeel, “Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search,” in ICRA, 2016. find
  36. S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” in NIPS Workshop on Learning, Inference and Control of Multi-Agent Systems, 2016. find
  37. N. Hemsoth, “The Next Wave of Deep Learning Applications,” Next Platform, September 2016. find
  38. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  39. T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for LVCSR,” in ICASSP, 2013. find
  40. V. Nair and G. E. Hinton, “Rectified Linear Units Improve Restricted Boltzmann Machines,” in ICML, 2010. find
  41. A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in ICML, 2013. find
  42. K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015. find
  43. D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),” ICLR, 2016.
  44. X. Zhang, J. Trmal, D. Povey, and S. Khudanpur, “Improving deep neural network acoustic models using generalized maxout networks,” in ICASSP, 2014. find
  45. Y. Zhang, M. Pezeshki, P. Brakel, S. Zhang, , C. Laurent, Y. Bengio, and A. Courville, “Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks,” in Interspeech, 2016. find
  46. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in ACM International Conference on Multimedia, 2014.
  47. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in ICML, 2015. find
  48. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov 1998. find
  49. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks,” in ICLR, 2014.
  50. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” in ICLR, 2015.
  51. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper With Convolutions,” in CVPR, 2015. find
  52. M. Lin, Q. Chen, and S. Yan, “Network in Network,” in ICLR, 2014. find
  53. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in CVPR, 2016. find
  54. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inceptionv4, Inception-ResNet and the Impact of Residual Connections on Learning,” in AAAI, 2017. find
  55. G. Urban, K. J. Geras, S. E. Kahou, O. Aslan, S. Wang, R. Caruana, A. Mohamed, M. Philipose, and M. Richardson, “Do Deep Convolutional Nets Really Need to be Deep and Convolutional?” ICLR, 2017. find
  56. “Caffe LeNet MNIST,” http://caffe.berkeleyvision.org/gathered/examples/mnist.html. ↗
  57. “Caffe Model Zoo,” http://caffe.berkeleyvision.org/model zoo. html. ↗
  58. “Matconvnet Pretrained Models,” http://www.vlfeat.org/matconvnet/pretrained/. ↗
  59. “TensorFlow-Slim image classification library,” https://github.com/tensorflow/models/tree/master/slim. ↗
  60. “Deep Learning Frameworks,” https://developer.nvidia.com/deep-learning-frameworks. ↗
  61. Y.-H. Chen, T. Krishna, J. Emer, and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” IEEE J. Solid-State Circuits, vol. 51, no. 1, 2017. find
  62. C. J. B. Yann LeCun, Corinna Cortes, “THE MNIST DATABASE of handwritten digits,” http://yann.lecun.com/exdb/mnist/. ↗
  63. L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in ICML, 2013. find
  64. A. Krizhevsky, V. Nair, and G. Hinton, “The CIFAR-10 dataset,” https://www.cs.toronto.edu/ ∼ kriz/cifar.html. ↗
  65. A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 11, pp. 1958–1970, 2008. find
  66. A. Krizhevsky and G. Hinton, “Convolutional deep belief networks on cifar-10,” Unpublished manuscript, vol. 40, 2010. find
  67. B. Graham, “Fractional max-pooling,” arXiv preprint arXiv:1412.6071, 2014. — [arXiv:1412.6071](https://arxiv.org/abs/1412.6071) find
  68. “Pascal VOC data sets,” http://host.robots.ox.ac.uk/pascal/VOC/. ↗
  69. “Microsoft Common Objects in Context (COCO) dataset,” http://mscoco.org/. ↗
  70. “Google Open Images,” https://github.com/openimages/dataset. ↗
  71. “YouTube-8M,” https://research.google.com/youtube8m/. ↗
  72. “AudioSet,” https://research.google.com/audioset/index.html. ↗
  73. S. Condon, “Facebook unveils Big Basin, new server geared for deep learning,” ZDNet, March 2017. find
  74. C. Dubout and F. Fleuret, “Exact acceleration of linear object detectors,” in ECCV, 2012. find
  75. J. Cong and B. Xiao, “Minimizing computation in convolutional neural networks,” in ICANN, 2014. find
  76. A. Lavin and S. Gray, “Fast algorithms for convolutional neural networks,” in CVPR, 2016.
  77. “Intel Math Kernel Library,” https://software.intel.com/en-us/mkl. ↗
  78. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cuDNN: Efficient Primitives for Deep Learning,” arXiv preprint arXiv:1410.0759, 2014.
  79. M. Horowitz, “Computing’s energy problem (and what we can do about it),” in ISSCC, 2014. find
  80. Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks,” in ISCA, 2016. find
  81. ——, “Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators,” IEEE Micro’s Top Picks from the Computer Architecture Conferences, vol. 37, no. 3, May-June 2017. find
  82. M. Sankaradas, V. Jakkula, S. Cadambi, S. Chakradhar, I. Durdanovic, E. Cosatto, and H. P. Graf, “A Massively Parallel Coprocessor for Convolutional Neural Networks,” in ASAP, 2009. find
  83. V. Sriram, D. Cox, K. H. Tsoi, and W. Luk, “Towards an embedded biologically-inspired machine vision processor,” in FPT, 2010. find
  84. S. Chakradhar, M. Sankaradas, V. Jakkula, and S. Cadambi, “A Dynamically Configurable Coprocessor for Convolutional Neural Networks,” in ISCA, 2010. find
  85. V. Gokhale, J. Jin, A. Dundar, B. Martini, and E. Culurciello, “A 240 G-ops/s Mobile Coprocessor for Deep Neural Networks,” in CVPR Workshop, 2014. find
  86. S. Park, K. Bong, D. Shin, J. Lee, S. Choi, and H.-J. Yoo, “A 1.93TOPS/W scalable deep learning/inference processor with tetra-parallel MIMD architecture for big-data applications,” in ISSCC, 2015. find
  87. L. Cavigelli, D. Gschwend, C. Mayer, S. Willi, B. Muheim, and L. Benini, “Origami: A Convolutional Network Accelerator,” in GLVLSI, 2015. find
  88. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep Learning with Limited Numerical Precision,” in ICML, 2015. find
  89. Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “ShiDianNao: Shifting Vision Processing Closer to the Sensor,” in ISCA, 2015. find
  90. M. Peemen, A. A. A. Setio, B. Mesman, and H. Corporaal, “Memory-centric accelerator design for Convolutional Neural Networks,” in ICCD, 2013. find
  91. C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks,” in FPGA, 2015. find
  92. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “DianNao: A Small-footprint High-throughput Accelerator for Ubiquitous Machine-learning,” in ASPLOS, 2014. find
  93. Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, “DaDianNao: A Machine-Learning Supercomputer,” in MICRO, 2014. find
  94. Y.-H. Chen, T. Krishna, J. Emer, and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” in ISSCC, 2016. find
  95. V. Sze, M. Budagavi, and G. J. Sullivan, “High Efficiency Video Coding (HEVC): Algorithms and Architectures,” in Integrated Circuit and Systems. Springer, 2014, pp. 1–375. find
  96. M. Alwani, H. Chen, M. Ferdman, and P. Milder, “Fused-layer CNN accelerators,” in MICRO, 2016. find
  97. D. Keitel-Schulz and N. Wehn, “Embedded DRAM development: Technology, physical design, and application issues,” IEEE Des. Test. Comput., vol. 18, no. 3, pp. 7–15, 2001. find
  98. J. Jeddeloh and B. Keeth, “Hybrid memory cube new DRAM architecture increases density and performance,” in Symp. on VLSI, 2012. find
  99. J. Standard, “High bandwidth memory (HBM) DRAM,” JESD235, 2013. find
  100. D. Kim, J. Kung, S. Chai, S. Yalamanchili, and S. Mukhopadhyay, “Neurocube: A programmable digital neuromorphic architecture with high-density 3D memory,” in ISCA, 2016. find
  101. M. Gao, J. Pu, X. Yang, M. Horowitz, and C. Kozyrakis, “TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory,” in ASPLOS, 2017. find
  102. J. Zhang, Z. Wang, and N. Verma, “A machine-learning classifier implemented in a standard 6T SRAM array,” in Symp. on VLSI, 2016. find
  103. Z. Wang, R. Schapire, and N. Verma, “Error-adaptive classifier boosting (EACB): Exploiting data-driven training for highly fault-tolerant hardware,” in ICASSP, 2014. find
  104. A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” in ISCA, 2016. find
  105. L. Chua, “Memristor-the missing circuit element,” IEEE Trans. Circuit Theory, vol. 18, no. 5, pp. 507–519, 1971. find
  106. L. Wilson, “International technology roadmap for semiconductors (ITRS),” Semiconductor Industry Association, 2013. find
  107. Lu, Darsen, “Tutorial on Emerging Memory Devices,” 2016. find
  108. S. B. Eryilmaz, S. Joshi, E. Neftci, W. Wan, G. Cauwenberghs, and H.-S. P. Wong, “Neuromorphic architectures with electronic synapses,” in ISQED, 2016. find
  109. P. Chi, S. Li, Z. Qi, P. Gu, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “PRIME: A Novel Processing-In-Memory Architecture for Neural Network Computation in ReRAM-based Main Memory,” in ISCA, 2016. find
  110. M. Prezioso, F. Merrikh-Bayat, B. Hoskins, G. Adam, K. K. Likharev, and D. B. Strukov, “Training and operation of an integrated neuromorphic network based on metal-oxide memristors,” Nature, vol. 521, no. 7550, pp. 61–64, 2015. find
  111. J. Zhang, Z. Wang, and N. Verma, “A matrix-multiplying ADC implementing a machine-learning classifier directly with data conversion,” in ISSCC, 2015. find
  112. E. H. Lee and S. S. Wong, “A 2.5 GHz 7.7 TOPS/W switched-capacitor matrix multiplier with co-designed local memory in 40nm,” in ISSCC, 2016. find
  113. R. LiKamWa, Y. Hou, J. Gao, M. Polansky, and L. Zhong, “RedEye: analog ConvNet image sensor architecture for continuous mobile vision,” in ISCA, 2016. find
  114. A. Wang, S. Sivaramakrishnan, and A. Molnar, “A 180nm CMOS image sensor with on-chip optoelectronic image compression,” in CICC, 2012. find
  115. H. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels,” in CVPR, 2016. find
  116. A. Suleiman and V. Sze, “Energy-efficient HOG-based object detection at 1080HD 60 fps with multi-scale support,” in SiPS, 2014. find
  117. E. H. Lee, D. Miyashita, E. Chai, B. Murmann, and S. S. Wong, “Lognet: Energy-Efficient Neural Networks Using Logrithmic Computations,” in ICASSP, 2017. find
  118. S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” in ICLR, 2016. find
  119. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” arXiv preprint arXiv:1609.07061, 2016.
  120. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160, 2016.
  121. Y. Ma, N. Suda, Y. Cao, J.-S. Seo, and S. Vrudhula, “Scalable and modularized RTL compilation of Convolutional Neural Networks onto FPGA,” in FPL, 2016. find
  122. P. Gysel, M. Motamedi, and S. Ghiasi, “Hardware-oriented Approximation of Convolutional Neural Networks,” in ICLR, 2016. find
  123. S. Higginbotham, “Google Takes Unconventional Route with Homegrown Machine Learning Chips,” Next Platform, May 2016. find
  124. T. P. Morgan, “Nvidia Pushes Deep Learning Inference With New Pascal GPUs,” Next Platform, September 2016. find
  125. P. Judd, J. Albericio, T. Hetherington, T. M. Aamodt, and A. Moshovos, “Stripes: Bit-serial deep neural network computing,” in MICRO, 2016. find
  126. B. Moons and M. Verhelst, “A 0.3–2.6 TOPS/W precision-scalable processor for real-time large-scale ConvNets,” in Symp. on VLSI, 2016. find
  127. M. Courbariaux, Y. Bengio, and J.-P. David, “Binaryconnect: Training deep neural networks with binary weights during propagations,” in NIPS, 2015.
  128. M. Courbariaux and Y. Bengio, “Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1,” arXiv preprint arXiv:1602.02830, 2016.
  129. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks,” in ECCV, 2016.
  130. Z. Cai, X. He, J. Sun, and N. Vasconcelos, “Deep learning with low precision by half-wave gaussian quantization,” in CVPR, 2017. find
  131. F. Li and B. Liu, “Ternary weight networks,” in NIPS Workshop on Efficient Methods for Deep Neural Networks, 2016. find
  132. C. Zhu, S. Han, H. Mao, and W. J. Dally, “Trained Ternary Quantization,” ICLR, 2017. find
  133. R. Andri, L. Cavigelli, D. Rossi, and L. Benini, “YodaNN: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights,” in ISVLSI, 2016. find
  134. K. Ando, K. Ueyoshi, K. Orimo, H. Yonekawa, S. Sato, H. Nakahara, M. Ikebe, T. Asai, S. Takamaeda-Yamazaki, and M. Kuroda, T.and Motomura, “BRein Memory: A 13-Layer 4.2 K Neuron/0.8 M Synapse Binary/Ternary Reconfigurable In-Memory Deep Neural Network Accelerator in 65nm CMOS,” in Symp. on VLSI, 201… find
  135. D. Miyashita, E. H. Lee, and B. Murmann, “Convolutional Neural Networks using Logarithmic Data Representation,” arXiv preprint arXiv:1603.01025, 2016. — [arXiv:1603.01025](https://arxiv.org/abs/1603.01025) find
  136. A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental Network Quantization: Towards Lossless CNNs with Low-precision Weights,” in ICLR, 2017. find
  137. W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing Neural Networks with the Hashing Trick,” in ICML, 2015. find
  138. J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, “Cnvlutin: ineffectual-neuron-free deep neural network computing,” in ISCA, 2016. find
  139. B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernández-Lobato, G.-Y. Wei, and D. Brooks, “Minerva: Enabling low-power, highly-accurate deep neural network accelerators,” in ISCA, 2016. find
  140. Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal Brain Damage,” in NIPS, 1990. find
  141. S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in NIPS, 2015. find
  142. T.-J. Yang, Y.-H. Chen, and V. Sze, “Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning,” in CVPR, 2017. find
  143. “DNN Energy Estimation,” http://eyeriss.mit.edu/energy.html. ↗
  144. R. Dorrance, F. Ren, and D. Marković, “A scalable sparse matrix-vector multiplication kernel for energy-efficient sparse-blas on FPGAs,” in ISFPGA, 2014. find
  145. S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “EIE: efficient inference engine on compressed deep neural network,” in ISCA, 2016.
  146. A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, “Scnn: An accelerator for compressed-sparse convolutional neural networks,” in ISCA, 2017. find
  147. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in NIPS, 2016.
  148. S. Anwar, K. Hwang, and W. Sung, “Structured pruning of deep convolutional neural networks,” ACM Journal of Emerging Technologies in Computing Systems, vol. 13, no. 3, p. 32, 2017. find
  149. J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke, “Scalpel: Customizing dnn pruning to the underlying hardware parallelism,” in ISCA, 2017. find
  150. H. Mao, S. Han, J. Pool, W. Li, X. Liu, Y. Wang, and W. J. Dally, “Exploring the regularity of sparse structure in convolutional neural networks,” in CVPR Workshop on Tensor Methods In Computer Vision, 2017. find
  151. J. S. Lim, “Two-dimensional signal and image processing,” Englewood Cliffs, NJ, Prentice Hall, 1990, 710 p., 1990. find
  152. F. Chollet, “Xception: Deep Learning With Depthwise Separable Convolutions,” CVPR, 2017. find
  153. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  154. F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \<1MB model size,” ICLR, 2017. find
  155. E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation,” in NIPS, 2014. find
  156. V. Lebedev, Y. Ganin, M. Rakhuba1, I. Oseledets, and V. Lempitsky, “Speeding-Up Convolutional Neural Networks Using Fine-tuned CP-Decomposition,” ICLR, 2015. find
  157. Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, “Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications,” in ICLR, 2016. find
  158. C. Bucilu, R. Caruana, and A. Niculescu-Mizil, “Model Compression,” in SIGKDD, 2006. find
  159. L. Ba and R. Caurana, “Do Deep Nets Really Need to be Deep?” NIPS, 2014. find
  160. G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” in NIPS Deep Learning Workshop, 2014.
  161. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for Thin Deep Nets,” ICLR, 2015. find
  162. “Benchmarking DNN Processors,” http://eyeriss.mit.edu/benchmarking.html. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sze-2017-efficient-processing-deep-neural-networks/