Elamur Research⌕
☰
home / papers / 2016 / DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Paper · 2016 · Shuchang Zhou, Yuxin Wu, Zekun Ni et al.

Summary

Large CNNs are costly to train and deploy. Earlier binarized models such as BNN and XNOR-Net reduced cost only in the forward pass. Their gradients stayed in full precision, so backpropagation still needed expensive floating-point convolutions. DoReFa-Net, from Megvii, generalizes binarization so that weights, activations and parameter gradients can each have an arbitrary low bitwidth. A dot product of low-bit fixed-point numbers can be split into bitwise AND and bitcount operations, with cost proportional to the product of the two bitwidths. This means bit convolution kernels on CPU, GPU, FPGA or ASIC could speed up both training and inference. Every quantizer is wrapped in a straight-through estimator. Weights are squashed with tanh, normalized by their maximum and rounded deterministically. Activations are bounded to [0, 1] and rounded. Gradients are rescaled per instance and quantized stochastically with added uniform noise. On SVHN and ImageNet the resulting networks come close to 32-bit baselines. An AlexNet variant with 1-bit weights, 2-bit activations and 6-bit gradients reaches 46.1% top-1. This suggests that cheap, energy-efficient low-bit training is feasible.

Key points

  • The AlexNet-derived DoReFa-Net with 1-bit weights, 2-bit activations and 6-bit gradients is trained from scratch. It achieves 46.1% single-crop top-1 accuracy on the ILSVRC12 validation set.
  • The model is released in TensorFlow format in the tensorpack repository, together with a reference training implementation for SVHN.
  • On SVHN, a configuration with 1-bit weights, 1-bit activations and 2-bit gradients reaches about 93% accuracy.
  • Gradients are the most sensitive to bitwidth, activations are less sensitive, and weights are the least. The method's name was chosen to reflect this ordering.
  • The authors state that no earlier work had quantized gradients below 8 bits while keeping accuracy comparable; prior low-precision training used convolutions with at least 10-bit numbers.
  • Binary weights are scaled by one constant shared across all filters, rather than XNOR-Net's per-channel factors. This keeps bit kernels usable when gradients are convolved with weights. The authors could not reproduce XNOR-Net's activation binarization results.

Interesting ideas

  • I1. Straight-through estimator as a general tool: the forward pass can apply any discrete operation, such as rounding or sampling, while the backward pass uses a hand-chosen surrogate gradient. This pattern could help in any pipeline with non-differentiable steps.
  • I2. Bit-plane decomposition of dot products: a multi-bit integer product becomes a weighted sum of 1-bit AND/bitcount products. Compute cost then scales directly with the bitwidths, which gives a tunable trade-off between precision and speed.
  • I3. Different quantization for different tensors: weights and activations can be rounded deterministically. Gradients, however, needed stochastic rounding plus uniform noise matched to the quantization step size. This is a useful lesson for anyone compressing gradients, for example to reduce communication in distributed training.
  • I4. Normalize unbounded tensors before quantizing: gradients are mapped into [0, 1] using a per-instance maximum, quantized, and then mapped back. Weights are first bounded with tanh. This is a simple recipe for quantizing values whose range is unknown in advance.
  • I5. Check the whole pipeline when designing compression: the authors dropped per-channel scaling because it would break bit kernels in the backward pass. This is a reminder that a compression scheme should be evaluated for training as well as inference, and that sweeping bitwidths separately for each component shows where precision really matters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen and Yuheng Zou. 2016. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv:1606.06160.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1606.06160 · arXiv 1606.06160

@misc{zhou2016dorefa,
  title = {{DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients}},
  author = {Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen and Yuheng Zou},
  year = {2016},
  doi = {10.48550/arxiv.1606.06160},
  eprint = {1606.06160},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.06160}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, Martın, Agarwal, Ashish, Barham, Paul, Brevdo, Eugene, Chen, Zhifeng, Citro, Craig, Corrado, Greg S, Davis, Andy, Dean, Jeffrey, Devin, Matthieu, et al. Tensorflow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow. org. find
  2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  3. Bengio, Yoshua, Léonard, Nicholas, and Courville, Aaron. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. find
  4. Chen, Tianshi, Du, Zidong, Sun, Ninghui, Wang, Jia, Wu, Chengyong, Chen, Yunji, and Temam, Olivier. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In ACM Sigplan Notices, volume 49, pp. 269–284. ACM, 2014a. find
  5. Chen, Yunji, Luo, Tao, Liu, Shaoli, Zhang, Shijin, He, Liqiang, Wang, Jia, Li, Ling, Chen, Tianshi, Xu, Zhiwei, Sun, Ninghui, et al. Dadiannao: A machine-learning supercomputer. In Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 609–622. IEEE Computer Societ… find
  6. Courbariaux, Matthieu and Bengio, Yoshua. Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
  7. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024, 2014. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  8. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. find
  9. CVPR 2009. IEEE Conference on, pp. 248–255. find
  10. Farabet, Clément, LeCun, Yann, Kavukcuoglu, Koray, Culurciello, Eugenio, Martini, Berin, Akselrod, Polina, and Talay, Selcuk. Large-scale fpga-based convolutional networks. Scaling up Machine Learning: Parallel and Distributed Approaches, pp. 399–419, 2011. find
  11. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  12. Gupta, Suyog, Agrawal, Ankur, Gopalakrishnan, Kailash, and Narayanan, Pritish. Deep learning with limited numerical precision. arXiv preprint arXiv:1502.02551, 2015. find
  13. Han, Song, Mao, Huizi, and Dally, William J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a. find
  14. Han, Song, Pool, Jeff, Tran, John, and Dally, William. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135–1143, 2015b. find
  15. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Mag… find
  16. Hinton, Geoffrey, Srivastava, Nitsh, and Swersky, Kevin. Neural networks for machine learning. Coursera, video lectures, 264, 2012b. find
  17. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  18. Kim, Minje and Smaragdis, Paris. Bitwise neural networks. arXiv preprint arXiv:1601.06071, 2016. — [arXiv:1601.06071](https://arxiv.org/abs/1601.06071) find
  19. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  20. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  21. Li, Fengfu and Liu, Bin. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016. — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) find
  22. Lin, Zhouhan, Courbariaux, Matthieu, Memisevic, Roland, and Bengio, Yoshua. Neural networks with few multiplications. arXiv preprint arXiv:1510.03009, 2015. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) find
  23. Merolla, Paul, Appuswamy, Rathinakumar, Arthur, John, Esser, Steve K, and Modha, Dharmendra. Deep neural networks are robust to weight binarization and other non-linear distortions. arXiv preprint arXiv:1606.01981, 2016. — [arXiv:1606.01981](https://arxiv.org/abs/1606.01981) find
  24. Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5. find
  25. Pham, Phi-Hung, Jelaca, Darko, Farabet, Clement, Martini, Berin, LeCun, Yann, and Culurciello, Eugenio. Neuflow: Dataflow vision processing system-on-a-chip. In Circuits and Systems (MWS-CAS), 2012 IEEE 55th International Midwest Symposium on, pp. 1044–1047. find
  26. Rastegari, Mohammad, Ordonez, Vicente, Redmon, Joseph, and Farhadi, Ali. Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279, 2016.
  27. Seide, Frank, Fu, Hao, Droppo, Jasha, Li, Gang, and Yu, Dong. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In INTERSPEECH, pp. 1058–1062, 2014. find
  28. Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, volume 1, 2011. find
  29. Wu, Jiaxiang, Leng, Cong, Wang, Yuhang, Hu, Qinghao, and Cheng, Jian. Quantized convolutional neural networks for mobile devices. arXiv preprint arXiv:1512.06473, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2016-dorefa-net-training-low-bitwidth/