Elamur Research⌕
☰
home / papers / 2015 / Deep Residual Learning for Image Recognition

Deep Residual Learning for Image Recognition

Paper · 2015 · Kaiming He, Xiangyu Zhang, Shaoqing Ren et al.

Summary

Plain convolutional networks show a puzzling degradation problem: once they are deep enough to converge, adding more layers raises training error. Overfitting therefore cannot explain it. The authors trace this to optimization difficulty. A deeper model could, in principle, copy a shallower one and make its extra layers identity mappings, yet standard SGD solvers fail to reach that solution. Their fix is residual learning. Each small stack of layers learns a correction F(x) that is added back to its input through an identity shortcut, rather than learning the whole target mapping. These shortcuts cost no extra parameters or computation, so plain and residual nets can be compared fairly. On ImageNet, residual nets up to 152 layers train easily and improve as depth grows. An ensemble reaches 3.57% top-5 test error and won ILSVRC 2015 classification. The same backbones also won the detection, localization and COCO tasks. This suggests that easing optimization through reparameterization, not only adding capacity, is what unlocks depth.

Key points

  • On CIFAR-10, a 56-layer plain net has higher training and test error than a 20-layer one. Residual connections remove this gap, and the paper explores CIFAR-10 models with 100 and 1000 layers.
  • The 152-layer ResNet is described as 8× deeper than VGG nets while still having lower complexity.
  • On ImageNet classification, the ensemble reaches 3.57% top-5 error, the 1st place result at ILSVRC 2015.
  • Swapping in the deeper representations alone yields a 28% relative improvement on COCO detection.
  • On ImageNet DET, ResNet-101 scores 58.8% mAP as a single model and 62.1% as an ensemble of 3, ahead of second place by 8.5 points.
  • On ImageNet localization, center-crop error with ground-truth classes drops from VGG's 33.1% to 13.3%. The test-set ensemble reaches 9.0% top-5 localization error, a 64% relative reduction over ILSVRC 14.

Interesting ideas

  • I1. Parameterize each block as "input plus a learned correction," so that doing nothing is the easy default and the model only has to learn deviations from it.
  • I2. The constructed-solution argument works as a diagnostic: a bigger model that can copy a smaller one should never fit training data worse, so a worse fit signals an optimizer problem rather than overfitting.
  • I3. Keep the architectural change cost-free (same parameters, depth, width and compute) so that the comparison isolates the idea itself.
  • I4. Measure how strongly each residual block actually responds; small responses show that most layers only lightly refine their input.
  • I5. Use "oracle" testing, feeding the ground-truth class to isolate localization quality from classification mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): self-induced repair rules can be framed as small deltas on top of a baseline parser's output, keeping each rule's effect local, explicit and easy to switch off.
  • I2 (learn): when adding a feature, stage or layer to gparse or the perceptron parser raises training error, treat it as a training bug to investigate rather than as a capacity limit.
  • I3 (learn): run ablations of pipeline variants at matched budgets, with the plain variant serving as a negative control.
  • I4 (tree): logging how much each gparse layer changes its representation could reveal which layers do real work and support the explainability goal.
  • I5 (answer): evaluate readers on gold UD trees and gold frames to separate reasoning errors from upstream parsing errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language processing · 131 works

Cite

Cite as Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1512.03385 · arXiv 1512.03385

@misc{he2015deep,
  title = {{Deep Residual Learning for Image Recognition}},
  author = {Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun},
  year = {2015},
  doi = {10.48550/arxiv.1512.03385},
  eprint = {1512.03385},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1512.03385}
}

References

50 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994. find
  2. C. M. Bishop. Neural networks for pattern recognition. Oxford university press, 1995. find
  3. W. L. Briggs, S. F. McCormick, et al. A Multigrid Tutorial. Siam, 2000. find
  4. K. Chatfield, V. Lempitsky, A. Vedaldi, and A. Zisserman. The devil is in the details: an evaluation of recent feature encoding methods. In BMVC, 2011. find
  5. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, pages 303–338, 2010. find
  6. S. Gidaris and N. Komodakis. Object detection via a multi-region & semantic segmentation-aware cnn model. In ICCV, 2015. find
  7. R. Girshick. Fast R-CNN. In ICCV, 2015. find
  8. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  9. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010. find
  10. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. arXiv:1302.4389, 2013. — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) find
  11. K. He and J. Sun. Convolutional neural networks at constrained time cost. In CVPR, 2015. find
  12. K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV, 2014. find
  13. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. find
  14. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012. find
  15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  16. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. find
  17. H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. TPAMI, 33, 2011. find
  18. H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P. Perez, and C. Schmid. Aggregating local image descriptors into compact codes. TPAMI, 2012. find
  19. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014.
  20. A. Krizhevsky. Learning multiple layers of features from tiny images. Tech Report, 2009. find
  21. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  22. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989. find
  23. Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998. find
  24. C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. arXiv:1409.5185, 2014. find
  25. M. Lin, Q. Chen, and S. Yan. Network in network. arXiv:1312.4400, 2013. — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  26. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV. 2014.
  27. J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. find
  28. G. Montúfar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014. find
  29. V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010. find
  30. F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In CVPR, 2007. find
  31. T. Raiko, H. Valpola, and Y. LeCun. Deep learning made easier by linear transformations in perceptrons. In AISTATS, 2012. find
  32. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
  33. S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object detection networks on convolutional feature maps. arXiv:1504.06066, 2015. find
  34. B. D. Ripley. Pattern recognition and neural networks. Cambridge university press, 1996. find
  35. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. find
  36. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014.
  37. A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120, 2013. find
  38. N. N. Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical report, 1998. find
  39. N. N. Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade, pages 207–226. Springer, 1998. find
  40. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. Le-Cun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR, 2014. find
  41. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  42. R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv:1505.00387, 2015.
  43. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. 1507.06228, 2015. find
  44. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. find
  45. R. Szeliski. Fast surface interpolation using hierarchical basis functions. TPAMI, 1990. find
  46. R. Szeliski. Locally adapted hierarchical basis preconditioning. In SIGGRAPH, 2006. find
  47. T. Vatanen, T. Raiko, H. Valpola, and Y. LeCun. Pushing stochastic gradient towards second-order methods–backpropagation learning with transformations in nonlinearities. In Neural Information Processing, 2013. find
  48. A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms, 2008. find
  49. W. Venables and B. Ripley. Modern applied statistics with s-plus. 1999. find
  50. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks. In ECCV, 2014. A. Object Detection Baselines find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/