Elamur Research
☰
home / papers / 2013 / Visualizing and Understanding Convolutional Networks

Visualizing and Understanding Convolutional Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1311.2901

Summary

Large convnets such as the Krizhevsky et al. (2012) ImageNet model classify images very well, but it is hard to see what their inner layers compute. Without that view, improving them comes down to trial and error. Zeiler and Fergus propose a diagnostic tool. A Deconvolutional Network (deconvnet) is attached to a trained convnet and maps any chosen activation back to pixel space. It does this by unpooling with recorded max-location "switches", rectifying, and applying the transposed filters. They add occlusion-based sensitivity tests and an ablation study of individual layers. Their model was trained on ImageNet 2012 (1.3M images, 1000 classes). The visualizations show that the features become more compositional, more invariant and more class-specific in higher layers. They also exposed defects in the first two layers, and fixing these gave an architecture that beats Krizhevsky et al. on ImageNet. When only the softmax is retrained, the features transfer well and set new records on Caltech-101 and Caltech-256. This work helped establish supervised ImageNet pre-training as a general-purpose feature extractor and made convnet interpretability a practical debugging method.

Key points

  • The deconvnet is not trained. It acts only as a probe that reuses the convnet's own filters (flipped) and the pooling switches recorded for a given input.
  • Training setup: SGD with batch 128, learning rate 10⁻², momentum 0.9, dropout 0.5 in layers 6–7, and filter RMS renormalization to radius 10⁻¹. Training ran 70 epochs, about 12 days on one GTX580.
  • Caltech-101 (30 images/class): 86.5% accuracy against the previous best of 84.3%. The same network trained from scratch reaches only 46.5%.
  • Caltech-256 (60 images/class): 74.2% against 55.2% for Bo et al. (2013). About 6 training images per class are enough to beat the previous best method, which used roughly ten times as many.
  • PASCAL 2012: mean accuracy is 79.0% against 82.2% for the leading method, but the model wins on 5 classes. Layer-wise SVM/softmax probes show accuracy rising steadily with depth, and the ablation shows that overall network depth matters more than any single layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

255 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 235 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1311.2901 · arXiv 1311.2901

@misc{zeiler2013visualizing,
  title = {{Visualizing and Understanding Convolutional Networks}},
  author = {Zeiler, Matthew D and Fergus, Rob},
  year = {2013},
  doi = {10.48550/arxiv.1311.2901},
  eprint = {1311.2901},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1311.2901}
}

References

23 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. Greedy layer-wise training of deep networks. In NIPS, pp. 153–160, 2007. find
  2. Berkes, P. and Wiskott, L. On the analysis and interpretation of inhomogeneous quadratic forms as receptive fields. Neural Computation, 2006. find
  3. Bo, L., Ren, X., and Fox, D. Multipath sparse coding using hierarchical matching pursuit. In CVPR, 2013. find
  4. Ciresan, D. C., Meier, J., and Schmidhuber, J. Multi-column deep neural networks for image classification. In CVPR, 2012. find
  5. Dalal, N. and Triggs, B. Histograms of oriented gradients for pedestrian detection. In CVPR, 2005. find
  6. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. DeCAF: A deep convolutional activation feature for generic visual recognition. In arXiv:1310.1531, 2013. find
  7. Erhan, D., Bengio, Y., Courville, A., and Vincent, P. Visualizing higher-layer features of a deep network. In Technical report, University of Montreal, 2009. find
  8. Fei-fei, L., Fergus, R., and Perona, P. One-shot learning of object categories. IEEE Trans. PAMI, 2006. find
  9. Griffin, G., Holub, A., and Perona, P. The caltech 256. In Caltech Technical Report, 2006. find
  10. Gunji, N., Higuchi, T., Yasumoto, K., Muraoka, H., Ushiku, Y., Harada, T., and Kuniyoshi, Y. Classification entry. In Imagenet Competition, 2012. find
  11. Hinton, G. E., Osindero, S., and The, Y. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006. find
  12. Hinton, G.E., Srivastave, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012. find
  13. Jarrett, K., Kavukcuoglu, K., Ranzato, M., and Le-Cun, Y. What is the best multi-stage architecture for object recognition? In ICCV, 2009. find
  14. Jianchao, Y., Kai, Y., Yihong, G., and Thomas, H. Linear spatial pyramid matching using sparse coding for image classification. In CVPR, 2009. find
  15. Krizhevsky, A., Sutskever, I., and Hinton, G.E. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. find
  16. Le, Q. V., Ngiam, J., Chen, Z., Chia, D., Koh, P., and Ng, A. Y. Tiled convolutional neural networks. In NIPS, 2010. find
  17. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Comput., 1(4):541–551, 1989. find
  18. Sande, K., Uijlings, J., Snoek, C., and Smeulders, A. Hybrid coding for selective search. In PASCAL VOC Classification Challenge 2012, 2012. find
  19. Sohn, K., Jung, D., Lee, H., and Hero III, A. Efficient learning of sparse, distributed, convolutional feature representations for object recognition. In ICCV, 2011. find
  20. Torralba, A. and Efros, A. A. Unbiased look at dataset bias. In CVPR, 2011. find
  21. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P. A. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096– 1103, 2008. find
  22. Yan, S., Dong, J., Chen, Q., Song, Z., Pan, Y., Xia, W., Huang, Z., Hua, Y., and Shen, S. Generalized hierarchical matching for sub-category aware object classification. In PASCAL VOC Classification Challenge 2012, 2012. find
  23. Zeiler, M., Taylor, G., and Fergus, R. Adaptive deconvolutional networks for mid and high level feature learning. In ICCV, 2011. find