Elamur Research⌕
☰
home / papers / 2016 / XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Paper · 2016 · Mohammad Rastegari, Vicente Ordonez, Joseph Redmon et al.

Summary

Modern CNNs demand far more memory and compute than phones or embedded hardware can supply. AlexNet, for example, needs about 249MB and roughly 1.5B high-precision operations per image. The authors propose two binary approximations of standard architectures, trained from scratch rather than compressed after pretraining. Binary-Weight-Networks (BWN) restrict filters to +1/−1 times a per-filter scaling factor, which cuts memory about 32× and turns convolutions into additions and subtractions. XNOR-Networks also binarize the inputs to convolutional layers, so convolutions reduce mostly to XNOR and bit-counting. That gives roughly 58× fewer high-precision operations and makes real-time inference on CPUs plausible. They evaluate on ImageNet (ILSVRC2012) with AlexNet, ResNet-18 and a GoogLenet variant. BWN AlexNet matches the full-precision model's accuracy, and both variants beat BinaryConnect and BinaryNet by more than 16% top-1. The work matters because it was the first evaluation of binarized networks at ImageNet scale, and it showed that extreme quantization need not destroy accuracy on hard tasks.

Key points

  • Efficiency claims: ~32× memory saving for both variants. Binary weights alone give ~2× speedup, while XNOR-Net gives ~58× fewer high-precision operations.
  • Comparison with prior binarization: on ILSVRC2012 the method beats BinaryNet by 16.3% top-1, and it stays ahead of BC and BNN by roughly 17% across all training epochs.
  • ResNet-18 results (top-1/top-5):
    • BWN: 60.8/83.0
    • XNOR-Net: 51.2/73.2
    • Full precision: 69.3/89.2
  • GoogLenet variant results (top-1/top-5): BWN reaches 65.5/86.1 versus 71.3/90.0 for full precision. This variant has 21 plain convolutional layers and no branching.
  • Ablations:
    • Computing α analytically gives 56.8 top-1, versus 46.2 when α is a learned layer.
    • Reordering the block from C-B-A-P to B-A-C-P lifts XNOR-Net top-1 from 30.3 to 44.2.
    • Dropping the input scaling β costs under 1% top-1.
    • Binary gradients cost only 1.4% top-1.

Interesting ideas

  • I1. Analytic scaling instead of learned scaling: pairing binary values with a per-filter magnitude derived in closed form each training iteration worked much better than letting the network learn that scalar. The same move could help other extreme-quantization settings, such as compressing embedding or transformer weights for on-device NLP models.
  • I2. Layer order as a quantization lever: placing normalization and binarization before the convolution cut the information lost to quantization and gave a large accuracy jump. When compressing any architecture, block ordering is worth testing, not just bit-width.
  • I3. Real-valued shadow weights: full-precision weights are kept and updated during training, while binarized copies are used in the forward pass. This is a simple, general recipe for training models that must run in low precision at deployment.
  • I4. Pruning cheap-to-remove components via ablation: the input scaling factor contributed little accuracy, so it can be dropped for extra speed. Measuring each approximation component separately shows which ones are worth their cost.
  • I5. Counting high-precision operations as the cost metric: reporting savings this way, rather than only wall-clock time, gives a hardware-independent cost measure. It is useful for comparing compressed models across different deployment targets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

Cite

Cite as Mohammad Rastegari, Vicente Ordonez, Joseph Redmon and Ali Farhadi. 2016. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. arXiv:1603.05279.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.05279 · arXiv 1603.05279

@misc{rastegari2016xnor,
  title = {{XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks}},
  author = {Mohammad Rastegari, Vicente Ordonez, Joseph Redmon and Ali Farhadi},
  year = {2016},
  doi = {10.48550/arxiv.1603.05279},
  eprint = {1603.05279},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.05279}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. (2012) 1097–1105 1, 10, 11, 12
  2. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) 1
  3. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1–9 1, 4, 11, 13 find
  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR (2015) 1, 4, 10, 11, 12
  5. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2014) 580–587 1
  6. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 1440–1448 1 find
  7. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems. (2015) 91–99 1
  8. Oculus, V.: Oculus rift-virtual reality headset for 3d gaming. URL: http://www.oculusvr. com (2012) 1 ↗
  9. Gottmer, M.: Merging reality and virtuality with microsoft hololens. (2015) 1 find
  10. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 3431–3440 2 find
  11. Courbariaux, M., Bengio, Y.: Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR (2016) 2, 3, 4, 6, 7, 10, 11, 12 find
  12. Denil, M., Shakibi, B., Dinh, L., de Freitas, N., et al.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems. (2013) 2148–2156 3 find
  13. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems 2(4) (1989) 303–314 3 find
  14. Seide, F., Li, G., Yu, D.: Conversational speech transcription using context-dependent deep neural networks. In: Interspeech. (2011) 437–440 3 find
  15. Dauphin, Y.N., Bengio, Y.: Big neural networks waste capacity. arXiv preprint arXiv:1301.3583 (2013) 3 — [arXiv:1301.3583](https://arxiv.org/abs/1301.3583) find
  16. Ba, J., Caruana, R.: Do deep nets really need to be deep? In: Advances in neural information processing systems. (2014) 2654–2662 3 find
  17. Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with backpropagation. In: Advances in neural information processing systems. (1989) 177–185 3 find
  18. LeCun, Y., Denker, J.S., Solla, S.A., Howard, R.E., Jackel, L.D.: Optimal brain damage. In: NIPs. Volume 89. (1989) 3 find
  19. Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann (1993) 3 find
  20. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems. (2015) 1135–1143 3 find
  21. Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015. Springer (2015) 677–684 3 find
  22. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015) 3 find
  23. Chen, W., Wilson, J.T., Tyree, S., Weinberger, K.Q., Chen, Y.: Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788 (2015) 3 — [arXiv:1504.04788](https://arxiv.org/abs/1504.04788) find
  24. Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in Neural Information Processing Systems. (2014) 1269–1277 3 find
  25. Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014) 3 find
  26. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013) 4 — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  27. Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR (2016) 4 find
  28. Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 1mb model size. arXiv preprint arXiv:1602.07360 (2016) 4 find
  29. Gong, Y., Liu, L., Yang, M., Bourdev, L.: Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 (2014) 4 — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  30. Arora, S., Bhaskara, A., Ge, R., Ma, T.: Provable bounds for learning some deep representations. arXiv preprint arXiv:1310.6343 (2013) 4 — [arXiv:1310.6343](https://arxiv.org/abs/1310.6343) find
  31. Vanhoucke, V., Senior, A., Mao, M.Z.: Improving the speed of neural networks on cpus. In: Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop. Volume 1. (2011) 4 find
  32. Hwang, K., Sung, W.: Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In: Signal Processing Systems (SiPS), 2014 IEEE Workshop on, IEEE (2014) 1–6 4 find
  33. Anwar, S., Hwang, K., Sung, W.: Fixed point optimization of deep convolutional neural networks for object recognition. In: Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, IEEE (2015) 1131–1135 4 find
  34. Lin, Z., Courbariaux, M., Memisevic, R., Bengio, Y.: Neural networks with few multiplications. arXiv preprint arXiv:1510.03009 (2015) 4 — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) find
  35. Courbariaux, M., Bengio, Y., David, J.P.: Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024 (2014) 4 — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  36. Soudry, D., Hubara, I., Meir, R.: Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights. In: Advances in Neural Information Processing Systems. (2014) 963–971 4 find
  37. Esser, S.K., Appuswamy, R., Merolla, P., Arthur, J.V., Modha, D.S.: Backpropagation for energy-efficient neuromorphic computing. In: Advances in Neural Information Processing Systems. (2015) 1117–1125 4 find
  38. Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems. (2015) 3105–3113 4, 6, 10, 11
  39. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Proceedings of the 30th International Conference on Machine Learning (ICML-13). (2013) 1058–1066 5 find
  40. Baldassi, C., Ingrosso, A., Lucibello, C., Saglietti, L., Zecchina, R.: Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses. Physical review letters 115(12) (2015) 128101 5 find
  41. Kim, M., Smaragdis, P.: Bitwise neural networks. arXiv preprint arXiv:1601.06071 (2016) 5 — [arXiv:1601.06071](https://arxiv.org/abs/1601.06071) find
  42. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 6, 12
  43. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015) 9, 11 find
  44. Redmon, J.: Darknet: Open source neural networks in c. http://pjreddie.com/darknet/ (2013–2016) 13 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rastegari-2016-xnor-net-imagenet-classification-using/