Elamur Research⌕
☰
home / papers / 2016 / Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Paper · 2016 · Sergey Zagoruyko and Nikos Komodakis

Summary

Knowledge transfer between CNNs usually means matching a teacher's outputs (knowledge distillation) or its intermediate features (FitNets). Zagoruyko and Komodakis instead transfer "attention": spatial maps showing which image regions a network relies on when deciding. A shallower or thinner student is trained with an extra loss that pulls its attention maps, taken at several depths, toward those of a stronger pretrained teacher. This adds no new weights. The authors study two kinds of maps. Activation-based maps aggregate absolute activation values across channels. Gradient-based maps come from gradients with respect to the input. Their visual analysis shows that more accurate networks have sharper, object-aligned attention. Mid-level maps highlight discriminative parts, while top layers cover whole objects. Experiments on CIFAR-10, an indoor Scenes dataset and ImageNet, with both residual and non-residual architectures, show consistent gains. Activation-based transfer outperforms matching full activations, and it can also be used alongside knowledge distillation. The method is a cheap, architecture-agnostic way to bring a strong model's behaviour into a smaller, faster one.

Key points

  • Activation-based attention maps use three channel-wise statistics of absolute activations: the plain sum, the sum of p-th powers, and the max of p-th powers (p > 1). Larger p concentrates weight on the most strongly activated regions.
  • For visualization, the authors compared Network-In-Network (62% top-1), ResNet-34 (73%) and ResNet-101 (77.3%). Stronger models show attention peaks that weaker ones lack.
  • On ImageNet, the attention transfer losses were added at epoch 60 of 100, with ResNet-34 as teacher. This cut ResNet-18's single-crop top-1/top-5 error from 30.4/10.8 to 29.3/10.0, against the teacher's 26.1/8.3.
  • On CIFAR-10, transfer from WRN-16-2 to WRN-16-1 sped up convergence and improved final accuracy. It also outperformed full-activation transfer (F-ActT).
  • Experiments ran in Torch, using fb.resnet.torch for ImageNet. Code and models are available at github.com/szagoruyko/attention-transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École nationale des ponts et chausséesChamps-sur-Marne · France4 works · 3,074 citations🔬Laboratoire d'Informatique Gaspard-MongeChamps-sur-Marne · France3 works · 1,590 citations🎓Paris-Est SupChamps-sur-Marne · France3 works · 1,590 citations🎓Pôle de recherche et d'enseignement supérieur Université Paris-EstChamps-sur-Marne · France3 works · 1,590 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Sergey Zagoruyko and Nikos Komodakis. 2016. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. arXiv:1612.03928.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1612.03928 · arXiv 1612.03928

@misc{zagoruyko2016paying,
  title = {{Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer}},
  author = {Sergey Zagoruyko and Nikos Komodakis},
  year = {2016},
  doi = {10.48550/arxiv.1612.03928},
  eprint = {1612.03928},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.03928}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. URL http://arxiv.org/abs/1409.0473. ↗
  2. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, pp. 535–541, 2006. find
  3. ImageNet→CUB 28.5 27 (-1.5) 27 (-1.5) 26.5 ImageNet→Scenes 28.2 28.1 (-0.1) 27.1 (-1.1) 26 Taco S. Cohen and Max Welling. Group equivariant convolutional networks. abs/1602.07576, 2016. URL http://arxiv.org/abs/1602.07576. ↗
  4. Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural Computation, 2012. find
  5. H. Drucker and Y LeCun. Improving generalization performance using double backpropagation. IEEE Transaction on Neural Networks, 3(6):991–997, 1992. find
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
  7. Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural networks. 2015. find
  8. Hugo Larochelle and Geoffrey E. Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 1243–1251. Curran Associates, Inc., 2… find
  9. Jimmy Ba Lei and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. find
  10. Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4400, 2013. — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  11. Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent models of visual attention. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2204–2212. Curran Associates, Inc., 2014. URL http:… find
  12. M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Is object localization for free? weakly-supervised learning with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. find
  13. O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In British Machine Vision Conference, 2015. find
  14. A. Quattoni and A. Torralba. Recognizing indoor scenes. In CVPR, 2009. find
  15. Ronald A. Rensink. The dynamic representation of scenes. In Visual Cognition, pp. 17–42, 2000. find
  16. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. Technical Report Arxiv report 1412.6550, arXiv, 2014. find
  17. Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. 2016. find
  18. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop, 2014.
  19. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In arXiv:1412.6806, also appeared at ICLR 2015 Workshop Track, 2015. URL http://arxiv.org/abs/1412.6806. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) ↗
  20. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015.
  21. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. find
  22. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044, 2015. URL http://arxiv.org/abs/1502.03044. ↗
  23. CoRR, Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. Stacked attention networks for image question answering. CoRR, abs/1511.02274, 2015. URL http://arxiv.org/abs/1511.02274. ↗
  24. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. find
  25. Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
  26. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Computer Vision and Pattern Recognition, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zagoruyko-2016-paying-more-attention-attention-improving/