Elamur Research⌕
☰
home / papers / 2012 / Multi-column Deep Neural Networks for Image Classification

Multi-column Deep Neural Networks for Image Classification

Paper · 2012 · Dan Cireşan, Ueli Meier and Juergen Schmidhuber

Summary

This paper addresses the gap between machine and human accuracy on visual recognition tasks such as reading handwritten digits or identifying traffic signs. The authors build deep convolutional networks (DNN) that alternate convolution and max-pooling layers with very small receptive fields, often 2x2 or 3x3, and place a standard multilayer perceptron on top. The networks are trained in a purely supervised way with online back-propagation on GPUs and do not use unsupervised pre-training. In every epoch, training images are randomly shifted, scaled and rotated, and character images are also elastically warped. Several such networks, called "columns", are trained on inputs that have been normalized or preprocessed differently, and their outputs are averaged to form a Multi-column DNN (MCDNN). The authors test this on MNIST, NIST SD 19, Chinese handwriting (HWDB1.0), traffic signs, CIFAR10 and NORB. It sets new records on all of them and lowers the best previous error rates by roughly 30–80%. The work is significant because it shows that large GPU-trained convolutional ensembles can match or even exceed human performance on vision benchmarks.

Key points

  • MNIST: The final MCDNN has 35 columns, made of 5 nets for each of 7 digit-width normalizations. It reaches 0.23% test error, which is close to the roughly 0.2% human error rate. A single net trained on all 7 preprocessed datasets reaches only 0.52%.
  • Ensembling and training cost: Combining columns cuts error by about 30–40% compared with individual DNNs. GPU code runs 50–100x faster than serial CPU code. One MNIST column takes about 14 hours to train.
  • Traffic signs: Error is 0.54%, compared with the previous best of 1.69%. This is about half the human error rate.
  • Other benchmarks: CIFAR10 error falls from 18.50% to 11.21%. On HWDB1.0, error is 5.61% for online handwriting and 6.5% for offline handwriting.
  • NORB (jittered-cluttered): A 4-column MCDNN reaches 3.57% error using 2 training folds. It reaches 2.70% using all 10 folds, compared with the earlier best of 5.00%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Dan Cireşan, Ueli Meier and Juergen Schmidhuber. 2012. Multi-column Deep Neural Networks for Image Classification. arXiv:1202.2745 · CVPR 2012, p. 3642-3649.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1202.2745 · arXiv 1202.2745

@misc{cirecsan2012multi,
  title = {{Multi-column Deep Neural Networks for Image Classification}},
  author = {Dan Cirecsan, Ueli Meier and Juergen Schmidhuber},
  year = {2012},
  doi = {10.48550/arxiv.1202.2745},
  eprint = {1202.2745},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1202.2745}
}

References

41 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Behnke. Hierarchical Neural Networks for Image Interpretation, volume 2766 of Lecture Notes in Computer Science. Springer, 2003. 1, 2 find
  2. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Neural Information Processing Systems, 2007. 1, 2 find
  3. N. P. Bichot, A. F. Rossi, and R. Desimone. Parallel and serial neural mechanisms for visual search in macaque area V4. Science, 308:529–534, 2005. 2 find
  4. P. R. Cavalin, A. de Souza Britto Jr., F. Bortolozzi, R. Sabourin, and L. E. S. de Oliveira. An implicit segmentation-based method for recognition of handwritten strings of characters. In SAC, pages 836– 840, 2006. 6 find
  5. D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep, big, simple neural nets for handwritten digit recognition. Neural Computation, 22(12):3207–3220, 2010. 1, 5 find
  6. D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Convolutional neural network committees for handwritten character classification. In International Conference on Document Analysis and Recognition, pages 1250–1254, 2011. 1, 5 find
  7. D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. Flexible, high performance convolutional neural networks for image classification. In International Joint Conference on Artificial Intelligence, pages 1237–1242, 2011. 1, 2, 4, 5, 8 find
  8. A. Coates and A. Y. Ng. The importance of encoding versus training with sparse coding and vector quantization. In International Conference on Machine Learning, 2011. 8, 10 find
  9. E. M. Dos Santos, L. S. Oliveira, R. Sabourin, and P. Maupin. Overfitting in the selection of classifier ensembles: a comparative study between pso and ga. In Conference on Genetic and Evolutionary Computation, pages 1423–1424. ACM, 2008. 6 MCDNN [%] 0.23 see Table 4 5.61 6.5 11.21 0.54 2.70 Relativ… find
  10. A. C. P.-A. M. P. V. Dumitru Erhan, Yoshua Bengio and S. Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625–660, 2010. 1, 2 find
  11. K. Fukushima. Neocognitron: A self-organizing neural network for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202, 1980. 1, 2 find
  12. E. Granger, P. Henniges, and R. Sabourin. Supervised Learning of Fuzzy ARTMAP Neural Networks Through Particle Swarm Optimization. Pattern Recognition, 1:27–60, 2007. 6 find
  13. P. J. Grother. NIST special database 19 - Handprinted forms and characters database. Technical report, National Institute of Standards and Thechnology (NIST), 1995. 1, 5 find
  14. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. 2 find
  15. D. H. Hubel and T. Wiesel. Receptive fields, binocular interaction, and functional architecture in the cat’s visual cortex. Journal of Physiology (London), 160:106–154, 1962. 2 find
  16. A. L. Koerich and P. R. Kalva. Unconstrained handwritten character recognition using metaclasses of characters. In Intl. Conf. on Image Processing, pages 542–545, 2005. 6 find
  17. T. Kohonen. Self-Organization and Associative Memory. Springer, second edition, 1988. 2 find
  18. A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Computer Science Department, University of Toronto, 2009. 1, 7 find
  19. Y. LeCun. Une procédure d’apprentissage pour réseau a seuil asymmetrique (a learning scheme for asymmetric threshold networks). In Proceedings of Cognitiva 85, pages 599–604, Paris, France, 1985. 2 find
  20. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. 1, 2, 3 find
  21. Y. LeCun, F.-J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. 1, 9 find
  22. Y. LeCun, L. D. Jackel, L. Bottou, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Sackinger, P. Simard, and V. Vapnik. Learning algorithms for classification: A comparison on handwritten digit recognition. In J. H. Oh, C. Kwon, and S. Cho, editors, Neural Networks: The Statistical M… find
  23. C.-L. Liu, F. Yin, D.-H. Wang, and Q.-F. Wang. Chinese Handwriting Recognition Contest. In Chinese Conference on Pattern Recognition, 2010. 1, 6, 7 find
  24. MATLAB. version 7.10.0 (R2010a). The MathWorks Inc., Natick, Massachusetts, 2010. 16 find
  25. J. Milgram, M. Cheriet, and R. Sabourin. Estimating accurate multi-class probabilities with support vector machines. In Int. Joint Conf. on Neural Networks, pages 1906–1911, 2005. 6 find
  26. M. Ranzato, Y.-L. B. Fu Jie Huang, and Y. LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proc. of Computer Vision and Pattern Recognition Conference, 2007. 2 find
  27. M. Ranzato, F. Huang, Y. Boureau, and Y. LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proc. Computer Vision and Pattern Recognition Conference (CVPR’07). IEEE Press, 2007. 5 find
  28. M. A. Ranzato, C. Poultney, S. Chopra, and Y. Lecun. Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems (NIPS 2006), 2006. 5 find
  29. M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nat. Neurosci., 2(11):1019–1025, 1999. 2 Technical Report No. IDSIA-04-12 find
  30. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pages 318–362. MIT Press, Cambridge, MA, USA, 1986. 2 find
  31. R. Salakhutdinov and G. Hinton. Learning a nonlinear embedding by preserving class neighborhood structure. In Proc. of the International Conference on Artificial Intelligence and Statistics, volume 11, 2007. 2 find
  32. D. Scherer, A. Müller, and S. Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In International Conference on Artificial Neural Networks, 2010. 10 find
  33. P. Sermanet and Y. LeCun. Traffic sign recognition with multi-scale convolutional networks. In Proceedings of International Joint Conference on Neural Networks (IJCNN’11), 2011. 17 find
  34. T. Serre, L. Wolf, S. M. Bileschi, M. Riesenhuber, and T. Poggio. Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell., 29(3):411–426, 2007. 2 find
  35. P. Y. Simard, D. Steinkraus, and J. C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Seventh International Conference on Document Analysis and Recognition, pages 958–963, 2003. 1, 2, 5 find
  36. J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In International Joint Conference on Neural Networks, 2011. 1, 7 find
  37. D. Strigl, K. Kofler, and S. Podlipnig. Performance and scalability of gpu-based convolutional neural networks. Parallel, Distributed, and Network-Based Processing, Euromicro Conference on, 0:317– 324, 2010. 1 find
  38. R. Uetz and S. Behnke. Large-scale object recognition with CUDA-accelerated hierarchical neural networks. In IEEE International Conference on Intelligent Computing and Intelligent Systems (ICIS), 2009. 1 find
  39. P. J. Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, 1974. 2 find
  40. D. H. Wiesel and T. N. Hubel. Receptive fields of single neurones in the cat’s striate cortex. J. Physiol., 148:574–591, 1959. 2 find
  41. D. J. Willshaw and C. von der Malsburg. How patterned neural connections can be set up by self-organization. Proc. R. Soc. London B, 194:431–445, 1976. 2 Technical Report No. IDSIA-04-12 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cire-csan-2012-multi-column-deep-neural-networks/