Elamur Research⌕
☰
home / papers / 2016 / Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Paper · 2016 · Matthieu Courbariaux, Itay Hubara, Daniel Soudry et al.

Summary

This work tackles the cost of running deep networks on low-power hardware. Modern DNNs depend on energy-hungry GPUs and full-precision arithmetic. The authors propose Binarized Neural Networks (BNNs), in which weights and hidden activations take only the values +1 or −1. These binary values are used both at inference and during the forward computation of gradients in training. Real-valued copies of the weights are kept only to accumulate small SGD updates. Gradients pass through the non-differentiable sign function via a saturating straight-through estimator. To remove most multiplications from training, the authors also introduce shift-based variants of batch normalization and AdaMax. Experiments in both Torch7 and Theano on MNIST, CIFAR-10 and SVHN reach accuracy close to the state of the art of the time. A custom binary matrix-multiplication GPU kernel runs the MNIST network 7 times faster than an unoptimized kernel, with no accuracy loss. This matters because it shows that whole networks can be reduced to bit-wise (XNOR/popcount) operations, which opens a path to very efficient dedicated hardware.

Key points

  • Two binarization schemes are used. The deterministic one is the sign function. The stochastic one outputs +1 with probability equal to a "hard sigmoid" of the input. The deterministic scheme is used almost everywhere, because random bits are expensive in hardware.
  • The gradient estimator through sign() is the incoming gradient multiplied by an indicator that |r| ≤ 1, which equals the derivative of hard tanh. The authors report that dropping this cancellation for large inputs clearly hurts performance.
  • Real-valued weights are clipped to [−1, 1] after each update. Without clipping they would drift without affecting their binary versions.
  • Shift-based BatchNorm and shift-based AdaMax replace multiplications with power-of-two approximations and bit shifts. The suggested AdaMax defaults are α = 2⁻¹⁰, 1−β₁ = 2⁻³ and 1−β₂ = 2⁻¹⁰. No accuracy loss was observed compared with vanilla BN.
  • Code for both frameworks is released as BinaryNet on GitHub. The authors also argue that repeated binary convolution kernels could let dedicated hardware cut time complexity by about 60%.

Interesting ideas

  • I1. Two copies of each weight. A low-precision copy is used for computation, while a high-precision "shadow" copy accumulates gradient updates. This is a general recipe for training any heavily quantized or discrete model, because tiny noisy SGD steps would otherwise be lost to rounding.
  • I2. Saturating straight-through estimator. The gradient passes through a hard threshold unchanged, but is zeroed where the input is already saturated. This cheap trick can be reused wherever a model contains discrete decisions, such as hashing layers or discrete codes.
  • I3. Quantization as regularization. The authors view binarization noise as a relative of Dropout and DropConnect. This suggests that compression and generalization need not be in tension, which is a useful framing when deciding how aggressively to compress a model.
  • I4. Bit-plane handling of non-binary input. The first layer receives 8-bit inputs, so the network splits them into 8 bit planes. It computes a binary dot product for each plane and sums the results weighted by powers of two. The approach keeps the whole pipeline binary-friendly, and similar decompositions could apply to other fixed-point inputs.
  • I5. Power-of-two approximations. Costly multiplications in normalization and optimizer updates are replaced with shifts based on the most significant bit. This is a hardware-friendly pattern applicable to edge deployment of many models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio. 2016. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv:1602.02830.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.02830 · arXiv 1602.02830

@misc{courbariaux2016binarized,
  title = {{Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1}},
  author = {Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1602.02830},
  eprint = {1602.02830},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.02830}
}

References

55 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. In ICLR’2015, arXiv:1409.0473, 2015.
  2. Baldassi, Carlo, Ingrosso, Alessandro, Lucibello, Carlo, Saglietti, Luca, and Zecchina, Riccardo. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12):1–5, 2015. find
  3. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
  4. Beauchamp, Michael J, Hauck, Scott, Underwood, Keith D, and Hemmert, K Scott. Embedded floating-point units in FPGAs. In Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays, pp. 12–20. find
  5. Bengio, Yoshua. Estimating or propagating gradients through stochastic neurons. Technical Report arXiv:1305.2982, Universite de Montreal, 2013. — [arXiv:1305.2982](https://arxiv.org/abs/1305.2982) find
  6. Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 20… find
  7. Chen, Tianshi, Du, Zidong, Sun, Ninghui, Wang, Jia, Wu, Chengyong, Chen, Yunji, and Temam, Olivier. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and opera… find
  8. Chen, Yunji, Luo, Tao, Liu, Shaoli, Zhang, Shijin, He, Liqiang, Wang, Jia, Li, Ling, Chen, Tianshi, Xu, Zhiwei, Sun, Ninghui, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pp. 609–622. find
  9. Cheng, Zhiyong, Soudry, Daniel, Mao, Zexi, and Lan, Zhenzhong. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015. — [arXiv:1503.03562](https://arxiv.org/abs/1503.03562) find
  10. Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with COTS HPC systems. In Proceedings of the 30th international conference on machine learning, pp. 1337–1345, 2013. find
  11. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011. find
  12. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Training deep neural networks with low precision multiplications. ArXiv e-prints, abs/1412.7024, December 2014. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  13. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Binaryconnect: Training deep neural networks with binary weights during propagations. ArXiv e-prints, abs/1511.00363, November 2015.
  14. Devlin, Jacob, Zbib, Rabih, Huang, Zhongqiang, Lamar, Thomas, Schwartz, Richard, and Makhoul, John. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL’2014, 2014.
  15. Dieleman, Sander, Schlter, Jan, Raffel, Colin, Olson, Eben, Snderby, Sren Kaae, Nouri, Daniel, Maturana, Daniel, Thoma, Martin, Battenberg, Eric, Kelly, Jack, Fauw, Jeffrey De, Heilman, Michael, diogo149, McFee, Brian, Weideman, Hendrik, takacsg84, peterderivaz, Jon, instagibbs, Rasul, Dr. Kashif, C… find
  16. Esser, Steve K, Appuswamy, Rathinakumar, Merolla, Paul, Arthur, John V, and Modha, Dharmendra S. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pp. 1117–1125, 2015. find
  17. Farabet, Clément, LeCun, Yann, Kavukcuoglu, Koray, Culurciello, Eugenio, Martini, Berin, Akselrod, Polina, and Talay, Selcuk. Large-scale FPGA-based convolutional networks. Machine Learning on Very Large Data Sets, 1, 2011a. find
  18. Farabet, Clément, Martini, Berin, Corda, Benoit, Akselrod, Polina, Culurciello, Eugenio, and LeCun, Yann. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pp. 109–116. find
  19. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In AIS-TATS’2010, 2010. find
  20. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing Deep Convolutional Networks using Vector Quantization. pp. 1–10. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  21. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout Networks. arXiv preprint, pp. 1319–1327. Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Frédéric, and Bengio, Y… — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) find
  22. Govindu, Gokul, Zhuo, Ling, Choi, Seonil, and Prasanna, Viktor. Analysis of high-performance floating-point arithmetic on FPGAs. In Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, pp. 149. find
  23. Graham, Benjamin. Spatially-sparse convolutional neural networks. arXiv preprint arXiv:1409.6070, 2014. — [arXiv:1409.6070](https://arxiv.org/abs/1409.6070) find
  24. Graves, Alex. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348–2356, 2011. find
  25. Han, Song, Pool, Jeff, Tran, John, and Dally, William. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135–1143, 2015. find
  26. Hinton, Geoffrey. Neural networks for machine learning. Coursera, video lectures, 2012. find
  27. Hinton, Geoffrey, Deng, Li, Dahl, George E., Mohamed, Abdelrahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and Kingsbury, Brian. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012. find
  28. Horowitz, Mark. Computing’s Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pp. 10–14, 2014. find
  29. Hwang, Kyuyeon and Sung, Wonyong. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pp. 1–6. find
  30. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015. find
  31. Judd, Patrick, Albericio, Jorge, Hetherington, Tayler, Aamodt, Tor, Jerger, Natalie Enright, Urtasun, Raquel, and Moshovos, Andreas. Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets. pp. 12. Kim, Jonghong, Hwang, Kyuyeon, and Sung, Wonyong. X1000 real-time phoneme recognition vlsi… find
  32. Kim, M. and Smaragdis, P. Bitwise Neural Networks. ArXiv eprints, January 2016. find
  33. Kim, Minje and Paris, Smaragdis. Bitwise Neural Networks. ICML Workshop on Resource-Efficient Machine Learning, 37, 2015. find
  34. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  35. Krizhevsky, A., Sutskever, I., and Hinton, G. ImageNet classification with deep convolutional neural networks. In NIPS’2012. 2012.
  36. LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. find
  37. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014. find
  38. Lee, Chen-Yu, Gallagher, Patrick W, and Tu, Zhuowen. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985, 2015. — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) find
  39. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network In Network. arXiv preprint, pp. 10. Lin, Zhouhan, Courbariaux, Matthieu, Memisevic, Roland, and Bengio, Yoshua. Neural networks with few multiplications. ArXiv e-prints, abs/1510.03009, October 2015. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) find
  40. Mnih, Volodymyr, Kavukcuoglo, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidgeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharsan, Wierstra, Daan, Legg, S… find
  41. Mordvintsev, Alexander, Olah, Christopher, and Tyka, Mike. In-ceptionism: Going deeper into neural networks, 2015. Accessed: 2015-06-30. find
  42. Pham, Phi-Hung, Jelaca, Darko, Farabet, Clement, Martini, Berin, LeCun, Yann, and Culurciello, Eugenio. Neuflow: dataflow vision processing system-on-a-chip. In Circuits and Systems (MWSCAS), 2012 IEEE 55th International Midwest Symposium on, pp. 1044–1047. find
  43. Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. find
  44. Sainath, Tara, rahman Mohamed, Abdel, Kingsbury, Brian, and Ramabhadran, Bhuvana. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013. find
  45. Silver, David, Huang, Aja, Maddison, Chris J., Guez, Arthur, Sifre, Laurent, van den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, Dieleman, Sander, Grewe, Dominik, Nham, John, Kalchbrenner, Nal, Sutskever, Ilya, Lillicrap, Timothy, Leach, Madele… find
  46. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  47. Soudry, Daniel, Di Castro, Dotan, Gal, Asaf, Kolodny, Avinoam, and Kvatinsky, Shahar. Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training. IEEE Transactions on Neural Networks and Learning Systems, (10): 2408–2421. Soudry, Daniel, Hubara, Itay, and Meir, Ron. Expectation… find
  48. Srivastava, Nitish. Improving neural networks with dropout. Master’s thesis, U. Toronto, 2013. find
  49. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
  50. Sung, Wonyong, Shin, Sungho, and Hwang, Kyuyeon. Resiliency of Deep Neural Networks under Quantization. (2014):1–9. find
  51. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In NIPS’2014, 2014.
  52. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014. find
  53. Tang, Yichuan. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013. find
  54. Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011. find
  55. Wan, Li, Zeiler, Matthew, Zhang, Sixin, LeCun, Yann, and Fergus, Rob. Regularization of neural networks using dropconnect. In ICML’2013, 2013. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/