Elamur Research⌕
☰
home / papers / 2016 / Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Paper · 2016 · Itay Hubara, Matthieu Courbariaux, Daniel Soudry et al.

Summary

The paper tackles the high cost of running deep networks, whose many multiply-accumulate operations make them slow and power-hungry on ordinary hardware and low-power devices. The authors train Quantized Neural Networks (QNNs), where weights and activations use very few bits. In the extreme case of 1 bit each, the result is a Binarized Neural Network (BNN). The quantized values are used both at inference and when computing gradients during training. Full-precision copies of the weights are kept only to accumulate updates. Gradients pass through the non-differentiable sign function via a straight-through estimator. The method is evaluated on MNIST, CIFAR-10, SVHN and ImageNet (AlexNet, GoogleNet), and on Penn Treebank language modeling with vanilla RNNs and LSTMs. Accuracy stays close to 32-bit baselines. The authors also write a binary matrix multiplication GPU kernel that runs an MNIST network substantially faster. This matters because most arithmetic becomes cheap bit-wise logic, which cuts memory traffic and should sharply reduce energy use.

Key points

  • A quantized AlexNet with 1-bit weights and 2-bit activations reaches about 51% top-1 accuracy on ImageNet.
  • Parameter gradients can be quantized to 6 bits with only minor accuracy loss, in preliminary experiments.
  • On Penn Treebank, recurrent QNNs (RNN and LSTM) with 4-bit weights and activations perform about as well as float32 models.
  • A custom binary GPU kernel runs the MNIST BNN 7× faster than an unoptimized kernel, with no accuracy drop. The authors also argue that dedicated hardware exploiting repeated binary convolution kernels could cut time complexity by about 60%.
  • Experiments were run in both Theano and Torch7, and the code is released as BinaryNet on GitHub (MatthieuCourbariaux and itayhubara repositories).

Interesting ideas

  • I1. Two copies of the weights. Binary weights are used in the forward and backward passes, while gradient updates accumulate in hidden real-valued weights. This separation could help train any model whose deployed parameters must be discrete, such as compact models for edge devices.
  • I2. Saturating straight-through estimator. The gradient passes through the sign function unchanged, but only when the input lies in [-1, 1]; outside that range it is set to zero. Real-valued weights are also clipped to the same range so they cannot drift without bound. This simple rule is reusable for other hard thresholds or discrete choices inside networks.
  • I3. Quantization as noise regularization. The authors present binarization as a Dropout-like noise source during training. Lowering precision may therefore help generalization rather than only costing accuracy.
  • I4. Replacing multiplications with shifts. Batch normalization, AdaMax and learning-rate decay are approximated with bit shifts, for example halving the learning rate via a 1-bit right shift every N epochs. This trick fits integer-only or very low-power training pipelines.
  • I5. Keep the output layer full-precision. The final classifier layer is not binarized. Leaving boundary layers in higher precision is a cheap design choice that protects accuracy in compressed models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States254 works · 56,747 citations🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel54 works · 7,237 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio. 2016. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. arXiv:1609.07061.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1609.07061 · arXiv 1609.07061

@misc{hubara2016quantized,
  title = {{Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations}},
  author = {Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1609.07061},
  eprint = {1609.07061},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.07061}
}

References

72 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Renzo Andri, Lukas Cavigelli, Davide Rossi, and Luca Benini. Yodann: An ultra-low power convolutional neural network accelerator based on binary weights. arXiv preprint arXiv:1606.05487, 2016. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR’2015, arXiv:1409.0473, 2015.
  3. Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12): 1–5, 2015. ISSN 10797114. doi: 10.1103/Phy… — [doi:10.1103/physrevlett.115.128101](https://doi.org/10.1103/physrevlett.115.128101) find
  4. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012
  5. Michael J Beauchamp, Scott Hauck, Keith D Underwood, and K Scott Hemmert. Embedded floating-point units in FPGAs. In Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays, pages 12–20. find
  6. Yoshua Bengio. Estimating or propagating gradients through stochastic neurons. Technical Report arXiv:1305.2982, Universite de Montreal, 2013. — [arXiv:1305.2982](https://arxiv.org/abs/1305.2982) find
  7. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral… find
  8. Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating sy… find
  9. Wenlin Chen, James T Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788, 2015. — [arXiv:1504.04788](https://arxiv.org/abs/1504.04788) find
  10. Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pages 609–622. find
  11. Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015. — [arXiv:1503.03562](https://arxiv.org/abs/1503.03562) find
  12. Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catanzaro, and Ng Andrew. Deep learning with COTS HPC systems. In Proceedings of the 30th international conference on machine learning, pages 1337–1345, 2013. find
  13. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011. find
  14. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. ArXiv e-prints, abs/1412.7024, December 2014. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  15. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. ArXiv e-prints, abs/1511.00363, November 2015a.
  16. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. Nips, pages 1–9, 2015b. URL http://arxiv.org/abs/1511.00363. ↗
  17. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL’2014, 2014.
  18. Sander Dieleman, Jan Schlter, Colin Raffel, Eben Olson, Sren Kaae Snderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, in-stagibbs, Dr. Kashif Rasul, CongLiu, Britef… — [doi:10.5281/zenodo.27878](https://doi.org/10.5281/zenodo.27878) find
  19. Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, and Dharmendra S Modha. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pages 1117–1125, 2015. find
  20. Clément Farabet, Yann LeCun, Koray Kavukcuoglu, Eugenio Culurciello, Berin Martini, Polina Akselrod, and Selcuk Talay. Large-scale FPGA-based convolutional networks. Machine Learning on Very Large Data Sets, 1, 2011a. find
  21. Clément Farabet, Berin Martini, Benoit Corda, Polina Akselrod, Eugenio Culurciello, and Yann LeCun. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pages 109–116. find
  22. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010, 2010. find
  23. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  24. Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frédéric Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013a. — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) find
  25. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout Networks. arXiv preprint, pages 1319–1327, 2013b. URL http://arxiv.org/abs/1302.4389. — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) ↗
  26. Gokul Govindu, Ling Zhuo, Seonil Choi, and Viktor Prasanna. Analysis of high-performance floating-point arithmetic on FPGAs. In Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, page 149. find
  27. Benjamin Graham. Spatially-sparse convolutional neural networks. arXiv:1409.6070, 2014. — [arXiv:1409.6070](https://arxiv.org/abs/1409.6070) find
  28. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348–2356, 2011. find
  29. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. CoRR, abs/1502.02551, 392, 2015. arXiv preprint Philipp Gysel, Mohammad Motamedi, and Soheil Ghiasi. Hardware-oriented approximation of convolutional neural networks. arXiv prep… find
  30. Huizi Mao Han, Song and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint, pages 1–11, 2015. URL http://arxiv.org/abs/1510.00149. ↗
  31. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a. find
  32. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135–1143, 2015b. find
  33. Geoffrey Hinton. Neural networks for machine learning. Coursera, video lectures, 2012. find
  34. Geoffrey Hinton, Li Deng, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012. find
  35. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  36. Mark Horowitz. Computing’s Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pages 10–14, 2014. ISSN 0018-9200. doi: 10.1109/JSSC.2014.2361354. — [doi:10.1109/jssc.2014.2361354](https://doi.org/10.1109/jssc.2014.2361354) find
  37. Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pages 1–6. find
  38. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015. find
  39. M. Kim and P. Smaragdis. Bitwise Neural Networks. ArXiv e-prints, January 2016. find
  40. Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], pages 1–13, 2014a. URL http://arxiv.org/abs/1412.6980$\\ delimiter"026E30F$nhttp://www.arxiv.org/pdf/1412.6980.pdf. ↗
  41. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014b.
  42. A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS’2012. 2012.
  43. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. find
  44. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014. find
  45. Chen-Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985, 2015. — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) find
  46. Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. ArXiv e-prints, abs/1510.03009, October 2015a. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) find
  47. Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural Networks with Few Multiplications. Iclr, pages 1–8, 2015b. URL http://arxiv.org/abs/1510.03009. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) ↗
  48. Chris Lomont. Fast inverse square root. Tech-315 nical Report, page 32, 2003. find
  49. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313– 330, 1993.
  50. Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve K Esser, and Dharmendra Modha. Deep neural networks are robust to weight binarization and other non-linear distortions. arXiv preprint arXiv:1606.01981, 2016. — [arXiv:1606.01981](https://arxiv.org/abs/1606.01981) find
  51. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT, pages 234–239, 2012. find
  52. Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01025, 2016. — [arXiv:1603.01025](https://arxiv.org/abs/1603.01025) find
  53. Volodymyr Mnih, Koray Kavukcuoglo, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidgeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharsan Kumaran, Daan Wierstra, Shane Legg, and Demis Ha… find
  54. Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html. Accessed: 2015-06-30. ↗
  55. Joachim Ott, Zhouhan Lin, Ying Zhang, Shih-Chii Liu, and Yoshua Bengio. Recurrent neural networks with limited numerical precision. arXiv preprint arXiv:1608.06902, 2016. — [arXiv:1608.06902](https://arxiv.org/abs/1608.06902) find
  56. Phi-Hung Pham, Darko Jelaca, Clement Farabet, Berin Martini, Yann LeCun, and Eugenio Culurciello. Neuflow: Dataflow vision processing system-on-a-chip. In Circuits and Systems (MWSCAS), 2012 IEEE 55th International Midwest Symposium on, pages 1044–1047. find
  57. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279, 2016.
  58. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. find
  59. Tara Sainath, Abdel rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013. find
  60. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukc… find
  61. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  62. Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS’2014, 2014. find
  63. H Spang and P Schultheiss. Reduction of quantizing noise by use of feedback. IRE Transactions on Communications Systems, 10(4):373–380, 1962. find
  64. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html. ↗
  65. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS’2014, 2014.
  66. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014. find
  67. Yichuan Tang. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013. find
  68. Naoya Torii, Hirotaka Kokubo, Dai Yamamoto, Kouichi Itoh, Masahiko Takenaka, and Tsutomu Matsumoto. Asic implementation of random number generators using sr latches and its evaluation. EURASIP Journal on Information Security, 2016(1):1–12, 2016. find
  69. Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011. find
  70. Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML’2013, 2013. find
  71. Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. pages 1984–1992, 2015. find
  72. Weiyi Zheng and Yina Tang. Binarized neural networks for language modeling. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hubara-2016-quantized-neural-networks-training-neural/