Elamur Research⌕
☰
home / papers / 2012 / ImageNet Classification with Deep Convolutional Neural Networks

ImageNet Classification with Deep Convolutional Neural Networks

Paper · 2012 · Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton

Summary

The authors worked on image classification from the ImageNet LSVRC-2010 contest. The training set contained about 1.2 million photographs in 1000 classes. They built a deep convolutional network (CNN) with five convolutional and three fully connected layers. The network has approximately 60 million parameters. To speed up training, the authors used the non-saturating ReLU activation. The model was split across two GTX 580 graphics cards. For this, they wrote their own optimized GPU implementation of convolutions (cuda-convnet) and released it publicly. To combat overfitting, they applied data augmentation and dropout in the fully connected layers. They also added local response normalization and overlapping pooling. On the 2010 test set, the top-1 error was 37.5% and the top-5 error was 17.0%, considerably better than the previous state of the art. One variant of the model won ILSVRC-2012 by a large margin. The work showed that a deep CNN trained purely with supervised learning outperforms methods based on hand-crafted features, provided there is plenty of labeled data and GPU computation. It was this work that launched the rapid spread of deep learning in computer vision.

Key points

  • Architecture: 8 learned layers, approximately 60 million parameters and 650 thousand neurons, with a 1000-way softmax at the output. Training took 5–6 days on two GTX 580 GPUs with 3 GB of memory each.
  • ReLU vs. tanh: on CIFAR-10, a four-layer network with ReLU reached 25% training error approximately six times faster.
  • Contribution of individual techniques:
    • response normalization reduces the top-1 and top-5 errors by 1.4 and 1.2 percentage points;
    • the two-GPU scheme reduces them by 1.7 and 1.2 percentage points;
    • removing any of the middle convolutional layers lowers top-1 accuracy by approximately 2%.
  • ILSVRC-2012: an ensemble of seven CNNs achieved 15.3% top-5 error on the test set, while second place (SIFT + FVs) had 26.2%. Two networks in the ensemble were first pre-trained on ImageNet Fall 2011 (15 million images, 22 thousand categories).
  • ImageNet Fall 2009 (10,184 categories, 8.9 million images): the top-1 and top-5 errors were 67.4% and 40.9%. The best previously published results were 78.1% and 60.9%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

💡 Concepts 3Key ideas and methods explained on our concept pages.Natural language processing · 131 worksMachine learning · 51 worksArtificial intelligence · 49 works

Cite

Cite as Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS 2012.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{krizhevsky2012imagenet,
  title = {{ImageNet Classification with Deep Convolutional Neural Networks}},
  author = {Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton},
  booktitle = {NeurIPS 2012},
  year = {2012},
  url = {https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html}
}

References

26 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R.M. Bell and Y. Koren. Lessons from the netflix prize challenge. ACM SIGKDD Explorations Newsletter, 9(2):75–79, 2007. find
  2. A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010. www.image-net.org/challenges. 2010. find
  3. L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001. find
  4. D. Cireşan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. Arxiv preprint arXiv:1202.2745, 2012. find
  5. D.C. Cireşan, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural networks for visual object classification. Arxiv preprint arXiv:1102.0183, 2011. — [arXiv:1102.0183](https://arxiv.org/abs/1102.0183) find
  6. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. find
  7. J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL http://www.image-net.org/challenges/LSVRC/2012/. ↗
  8. L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59–70, 2007. find
  9. G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, California Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694. ↗
  10. G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. find
  11. K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In International Conference on Computer Vision, pages 2146–2153. IEEE, 2009. find
  12. A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009. find
  13. A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010. find
  14. A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval. In ESANN, 2011. find
  15. Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990. find
  16. Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004. find
  17. Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253–256. IEEE, 2010. find
  18. H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616. ACM, 2009. find
  19. T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer Vision, Florence, Italy, October 2012. find
  20. V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning, 2010. find
  21. N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computational biology, 4(1):e27, 2008. find
  22. N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579, 2009. find
  23. B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173, 2008. find
  24. J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665–1672. IEEE, 2011. find
  25. P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003. find
  26. S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2):511–538, 2010. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/