Elamur Research⌕
☰
home / papers / 2015 / BinaryConnect: Training Deep Neural Networks with binary weights during propagations

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Paper · 2015 · Matthieu Courbariaux, Yoshua Bengio and Jean-Pierre David

Summary

Multipliers are the most area- and energy-hungry parts of digital neural network hardware. This paper asks whether a deep network can be trained while its weights take only two values (+1 or −1) during computation, so that most multiply-accumulate operations become additions and subtractions. BinaryConnect binarizes the weights for the forward and backward passes, either deterministically with the sign function or stochastically with a hard-sigmoid probability. The gradient updates still go into stored real-valued weights, which are clipped to [−1, 1]. The authors treat the binarization noise as a regularizer, similar to DropConnect. They test the method on permutation-invariant MNIST, CIFAR-10 (a VGG-like CNN, no data augmentation) and SVHN, and report results close to the state of the art. The stochastic variant behaves as a regularizer, though somewhat weaker than Dropout on MNIST. This matters for specialized low-power hardware. Training could skip about two-thirds of multiplications. Deterministic binary weights at inference could remove multiplications entirely and cut weight memory at least 16-fold.

Key points

  • Weights are binarized only in the forward and backward propagations. The parameter update keeps full precision, because SGD needs fine resolution to accumulate many tiny, noisy gradient steps (earlier work cited here found SGD needs at least 6–8 bits).
  • There are two binarization modes:
    • Deterministic: sign(w).
    • Stochastic: +1 with probability given by the hard sigmoid clip((w+1)/2, 0, 1), which keeps the expected value of the weight.
  • The CIFAR-10 architecture is (2×128C3)−MP2−(2×256C3)−MP2−(2×512C3)−MP2−(2×1024FC)−10SVM.
    • It is trained with square hinge loss, ADAM, Batch Normalization (minibatch 50) and an exponentially decaying learning rate, for 500 epochs.
    • SVHN uses half the hidden units and 200 epochs.
  • In an ablation on a small CIFAR-10 CNN (Table 1), test error depended on the optimizer (SGD, Nesterov momentum, ADAM) and on whether learning rates were scaled by the initialization coefficients. Errors ranged from 15.65% down to 10.47%.
  • Code is released at github.com/MatthieuCourbariaux/BinaryConnect. Experiments used Theano, Pylearn2 and Lasagne.

Interesting ideas

  • I1. Separate the computing copy from the accumulating copy. A cheap, coarse version of the parameters is used for computation, while a precise hidden copy collects the updates. This pattern can carry over to other quantized or low-resource training setups where only the expected value has to stay precise.
  • I2. Treat quantization as noise injection. Viewing discretization as an unbiased random corruption, in the same family as Dropout and DropConnect, turns a hardware constraint into a possible source of regularization. This view could help when compressing models trained on small datasets.
  • I3. Clip the latent weights to the range that affects the output. Once a real-valued weight moves past ±1, its binary version no longer changes. Clipping keeps the stored weights from drifting uselessly, which is a simple rule for any straight-through-style discretization.
  • I4. Use cheap piecewise-linear functions in place of smooth ones. The hard sigmoid replaces the logistic function for sampling probabilities. It costs less in software and hardware and still worked well in the experiments.
  • I5. Choose the inference mode separately from training. The authors compare using binary weights, using real weights, or averaging an ensemble of sampled binary networks. This is a reusable evaluation habit: decide whether a trick is meant to speed up training, inference, or both.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🔬Canadian Institute for Advanced ResearchToronto · Canada35 works · 19,316 citations🎓Polytechnique MontréalMontreal · Canada11 works · 35,142 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Matthieu Courbariaux, Yoshua Bengio and Jean-Pierre David. 2015. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. arXiv:1511.00363.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.00363 · arXiv 1511.00363

@misc{courbariaux2015binaryconnect,
  title = {{BinaryConnect: Training Deep Neural Networks with binary weights during propagations}},
  author = {Matthieu Courbariaux, Yoshua Bengio and Jean-Pierre David},
  year = {2015},
  doi = {10.48550/arxiv.1511.00363},
  eprint = {1511.00363},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.00363}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Geoffrey Hinton, Li Deng, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012. find
  2. Tara Sainath, Abdel rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013. find
  3. A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS’2012. 2012.
  4. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014. find
  5. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL’2014, 2014.
  6. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS’2014, 2014.
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR’2015, arXiv:1409.0473, 2015.
  8. Rajat Raina, Anand Madhavan, and Andrew Y. Ng. Large-scale deep unsupervised learning using graphics processors. In ICML’2009, 2009. find
  9. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, 2003.
  10. J. Dean, G.S Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS’2012, 2012. find
  11. Sang Kyun Kim, Lawrence C McAfee, Peter Leonard McMahon, and Kunle Olukotun. A highly scalable restricted Boltzmann machine FPGA implementation. In Field Programmable Logic and Applications, 2009. FPL 2009. International Conference on, pages 367–372. IEEE, 2009. find
  12. Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating sy… find
  13. Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pages 609–622. IEEE, 2014. find
  14. Lorenz K Muller and Giacomo Indiveri. Rounding methods for neural networks with low resolution synaptic weights. arXiv preprint arXiv:1504.05767, 2015. — [arXiv:1504.05767](https://arxiv.org/abs/1504.05767) find
  15. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML’2015, 2015. find
  16. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Low precision arithmetic for deep learning. In Arxiv:1412.7024, ICLR’2015 Workshop, 2015. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  17. Thomas M Bartol, Cailey Bromer, Justin P Kinney, Michael A Chirillo, Jennifer N Bourne, Kristen M Harris, and Terrence J Sejnowski. Hippocampal spine head sizes are highly precise. bioRxiv, 2015. find
  18. Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348–2356. Curran Associates, Inc., 2011. find
  19. Nitish Srivastava. Improving neural networks with dropout. Master’s thesis, U. Toronto, 2013. find
  20. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
  21. Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML’2013, 2013. find
  22. J.P. David, K. Kalach, and N. Tittley. Hardware complexity of modular multiplication and exponentiation. Computers, IEEE Transactions on, 56(10):1308–1319, Oct 2007. find
  23. R. Collobert. Large Scale Machine Learning. PhD thesis, Université de Paris VI, LIP6, 2004. find
  24. X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In AISTATS’2011, 2011. find
  25. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010, 2010. find
  26. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015. find
  27. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  28. Yu Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o(1/k 2 ). Doklady AN SSSR (translated as Soviet. Math. Docl.), 269:543–547, 1983. find
  29. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. Technical Report Arxiv report 1302.4389, Université de Montréal, February 2013. find
  30. Yichuan Tang. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013. find
  31. Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013. — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  32. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014. find
  33. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. find
  34. V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML’2010, 2010. find
  35. Benjamin Graham. Spatially-sparse convolutional neural networks. arXiv preprint arXiv:1409.6070, 2014. — [arXiv:1409.6070](https://arxiv.org/abs/1409.6070) find
  36. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  37. Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS’2014, 2014. find
  38. Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015. — [arXiv:1503.03562](https://arxiv.org/abs/1503.03562) find
  39. Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pages 1–6. IEEE, 2014. find
  40. Jonghong Kim, Kyuyeon Hwang, and Wonyong Sung. X1000 real-time phoneme recognition vlsi using feed-forward deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 7510–7514. IEEE, 2014. find
  41. Thomas P Minka. Expectation propagation for approximate bayesian inference. In UAI’2001, 2001. find
  42. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral… find
  43. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
  44. Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frédéric Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013. — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) find
  45. Sander Dieleman, Jan Schlter, Colin Raffel, Eben Olson, Sren Kaae Snderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, instagibbs, Dr. Kashif Rasul, CongLiu, Britefu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/