Elamur Research⌕
☰
home / papers / 2016 / EIE: Efficient Inference Engine on Compressed Deep Neural Network

EIE: Efficient Inference Engine on Compressed Deep Neural Network

Paper · 2016 · Song Han, Xingyu Liu, Huizi Mao et al.

Summary

Large neural networks spend most of their energy fetching weights from off-chip DRAM rather than on arithmetic. A DRAM access costs about 128 times more than an on-chip SRAM read, which makes such models hard to run on battery-powered devices. Fully connected layers suffer most because their weights are not reused, and batching to amortise memory traffic adds unacceptable latency in real-time settings. The authors build EIE, a custom ASIC accelerator that runs directly on models already shrunk by Deep Compression (pruning plus weight sharing), so networks like AlexNet fit entirely in SRAM. EIE is an array of processing elements. Each element stores a slice of the sparse weight matrix, skips work for both pruned weights and zero input activations, and decodes 4-bit weight indices through a small table of shared values. On nine CNN and LSTM benchmarks, EIE beats uncompressed CPU and GPU baselines by 189× and 13× in speed and is thousands of times more energy-efficient. The work shows that model compression delivers its full benefit only when the hardware is designed around the compressed format.

Key points

  • The energy gains come from four sources, as the authors attribute them:
    • roughly 120× from keeping weights in SRAM instead of DRAM;
    • about 10× from weight sparsity;
    • 8× from weight sharing;
    • about 3× from skipping zero ReLU outputs.
  • After pruning, the benchmark layers keep only 4–25% of their weights. Each surviving weight is a 4-bit pointer into a table of 16 shared values.
  • Each PE in 45nm CMOS occupies 0.638 mm² and draws 9.16 mW at 800 MHz. It holds 131K compressed weights, which correspond to about 1.2M dense ones.
  • Sixty-four PEs (40.8 mm²) handle the AlexNet FC layers at 1.88×10⁴ frames/s using under 600 mW. The design delivers 102 GOPS/s on the compressed model, equivalent to roughly 3 TOPS/s on a dense one.
  • Against the baselines:
    • Intel i7-5930k CPU: 24,000× better energy efficiency.
    • GeForce TITAN X GPU: 3,400× better energy efficiency.
    • Tegra K1 mobile GPU: 2,700× better energy efficiency.
    • DaDianNao: 2.9× higher throughput, 19× better energy efficiency, 3× better area efficiency.

Interesting ideas

  • I1. Use both kinds of sparsity. EIE skips a multiply only when the weight is nonzero and the incoming activation is also nonzero, rather than relying on fixed weight sparsity alone. About 70% of activations are zero in typical workloads, so this input-dependent zero-skipping yields large extra savings. The idea transfers to any sparse inference kernel.
  • I2. Store weights as codebook indices. Each weight is a tiny index into a short table of shared values rather than a full number. This trick could shrink storage for embedding tables or language models running on low-resource devices.
  • I3. Interleave rows across processing units. Matrix rows and output elements are distributed round-robin among the PEs, so each unit gets a mix of the irregular sparse work. This is a simple way to balance load and keep scaling good in any parallel sparse workload.
  • I4. Do the energy arithmetic first. The paper motivates its design with a quick estimate: a 1G-connection model at 20 Hz would need about 12.8 W for DRAM traffic alone. Such rough calculations are a cheap way to tell whether a deployment target is realistic before building anything.
  • I5. Report savings per technique. The paper credits each technique with its own multiplier and states throughput in both compressed and dense-equivalent terms. This makes it clear which design choice matters most and keeps comparisons with dense systems honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢NvidiaSanta Clara · United States37 works · 12,656 citations🏢Nvidia (United Kingdom)Reading · United Kingdom27 works · 11,141 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz and William J. Dally. 2016. EIE: Efficient Inference Engine on Compressed Deep Neural Network. arXiv:1602.01528.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.01528 · arXiv 1602.01528

@misc{han2016eie,
  title = {{EIE: Efficient Inference Engine on Compressed Deep Neural Network}},
  author = {Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz and William J. Dally},
  year = {2016},
  doi = {10.48550/arxiv.1602.01528},
  eprint = {1602.01528},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.01528}
}

References

45 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NIPS, 2012.
  2. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” arXiv:1409.4842, 2014. find
  3. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556, 2014.
  4. T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, “Recurrent neural network based language model.” in INTER-SPEECH, September 26-30, 2010, 2010, pp. 1045–1048. find
  5. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. find
  6. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in CVPR. IEEE, 2014, pp. 1701–1708. find
  7. A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” arXiv:1412.2306, 2014.
  8. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew, “Deep learning with cots hpc systems,” in 30th ICML, 2013. find
  9. M. Horowitz. Energy table for 45nm process, Stanford VLSI wiki. [Online]. Available: https://sites.google.com/site/seecproject ↗
  10. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “Diannao: a small-footprint high-throughput accelerator for ubiquitous machine-learning,” in ASPLOS, 2014. find
  11. Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, “Dadiannao: A machine-learning supercomputer,” in MICRO, December 2014. find
  12. Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “Shidiannao: shifting vision processing closer to the sensor,” in ISCA. ACM, 2015, pp. 92–104. find
  13. C. Farabet, C. Poulet, J. Y. Han, and Y. LeCun, “Cnp: An fpga-based processor for convolutional networks,” in FPL, 2009. find
  14. J. Qiu, J. Wang, S. Yao, K. Guo, B. Li, E. Zhou, J. Yu, T. Tang, N. Xu, S. Song, Y. Wang, and H. Yang, “Going deeper with embedded fpga platform for convolutional neural network,” in FPGA, 2016. find
  15. A. Shafiee and et al., “ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,” ISCA, 2016. find
  16. S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in Proceedings of Advances in Neural Information Processing Systems, 2015. find
  17. R. Girshick, “Fast R-CNN,” arXiv:1504.08083, 2015. find
  18. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, 1997. find
  19. A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural Networks, 2005. find
  20. N. D. Lane and P. Georgiev, “Can deep learning revolutionize mobile sensing?” in International Workshop on Mobile Computing Systems and Applications. ACM, 2015, pp. 117–122. find
  21. Richard Dorrance and Fengbo Ren and Dejan Marković, “A Scalable Sparse Matrix-vector Multiplication Kernel for Energy-efficient Sparse-blas on FPGAs,” in FPGA, 2014. find
  22. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in ICML, 2010. find
  23. S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” International Conference on Learning Representations 2016. find
  24. R. W. Vuduc, “Automatic performance tuning of sparse matrix kernels,” Ph.D. dissertation, UC Berkeley, 2003. find
  25. N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “Cacti 6.0: A tool to model large caches,” HP Laboratories, pp. 22–31, 2009. find
  26. NVIDIA. Technical brief: NVIDIA jetson TK1 development kit bringing GPU-accelerated computing to embedded systems. find
  27. NVIDIA. Whitepaper: GPU-based deep learning inference: A performance and power analysis. find
  28. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” arXiv:1408.5093, 2014.
  29. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition. 2009. find
  30. F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5mb model size,” arXiv:1602.07360, 2016. find
  31. Ling Zhuo and Viktor K. Prasanna, “Sparse Matrix-Vector Multiplication on FPGAs,” in FPGA, 2005. find
  32. V. Eijkhout, LAPACK working note 50: Distributed sparse data structures for linear algebra operations, 1992. find
  33. A. Lavin, “Fast algorithms for convolutional neural networks,” arXiv:1509.09308, 2015.
  34. S. J. Hanson and L. Y. Pratt, “Comparing biases for minimal network construction with back-propagation,” in NIPS, 1989. find
  35. Y. LeCun, J. S. Denker, S. A. Solla, R. E. Howard, and L. D. Jackel, “Optimal brain damage.” in NIPs, vol. 89, 1989. find
  36. B. Hassibi, D. G. Stork et al., “Second order derivatives for network pruning: Optimal brain surgeon,” Advances in neural information processing systems, pp. 164–164, 1993. find
  37. E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in NIPS 2014. find
  38. X. Zhang, J. Zou, X. Ming, K. He, and J. Sun, “Efficient and accurate approximations of nonlinear convolutional networks,” arXiv:1411.4229, 2014. find
  39. B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernndez-Lobato, G.-Y. Wei, and D. Brooks, “Minerva: Enabling low-power, highly-accurate deep neural network accelerators,” ISCA, 2016. find
  40. S. K. Esser and et al., “Convolutional networks for fast, energy-efficient neuromorphic computing,” arXiv:1603.08270, 2016. — [arXiv:1603.08270](https://arxiv.org/abs/1603.08270) find
  41. C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing fpga-based accelerator design for deep convolutional neural networks,” in FPGA, 2015. find
  42. Alexander Monakov and Anton Lokhmotov and Arutyun Avetisyan, “Automatically tuning sparse matrix-vector multiplication for GPU architectures,” in HiPEAC, 2010. find
  43. N. Bell and M. Garland, “Efficient sparse matrix-vector multiplication on cuda,” Nvidia Technical Report NVR-2008-004, Tech. Rep., 2008. find
  44. Bell, Nathan and Garland, Michael, “Implementing Sparse Matrix-vector Multiplication on Throughput-oriented Processors,” in High Performance Computing Networking, Storage and Analysis, 2009. find
  45. J. Fowers and K. Ovtcharov and K. Strauss and E.S. Chung and G. Stitt, “A high memory bandwidth fpga accelerator for sparse matrix-vector multiplication,” in FCCM, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2016-eie-efficient-inference-engine-compressed/