Elamur Research
☰
home / papers / 2020 / Training data-efficient image transformers & distillation through attention

Training data-efficient image transformers & distillation through attention

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2012.12877

Summary

Vision Transformer (ViT) models previously reached strong ImageNet accuracy only after pre-training on huge private collections such as JFT-300M, with heavy compute. This made them hard for most groups to adopt. The authors show that the same convolution-free architecture can be competitive when trained on Imagenet1k alone. Their Data-efficient image Transformers (DeiT) train on a single 8-GPU machine in about two to three days. The gains come from a carefully tuned training recipe: strong augmentation, regularization, AdamW with a lower weight decay, and especially repeated augmentation. They also propose a distillation method designed for transformers. An extra learnable "distillation token" sits alongside the class token, interacts with the image patches through attention, and is trained to match the teacher's predicted label. A convnet teacher works better than a transformer teacher. The distilled DeiT models reach up to 85.2% top-1 on ImageNet and transfer well to downstream benchmarks. This matters because high-quality vision transformers become feasible without massive datasets. Code and models are publicly released.

Key points

  • DeiT-B (86M parameters, same as ViT-B): reaches 83.1% top-1 single-crop on ImageNet with no external data. This uses 384×384 fine-tuning; at 224 it scores 81.8%. Pre-training takes 53 hours, plus an optional 20 hours of fine-tuning on one 8-GPU node.
  • Smaller variants: DeiT-S and DeiT-Ti are positioned as transformer counterparts of ResNet-50 and ResNet-18. They train in under 3 days on 4 GPUs.
  • Distillation token (DeiT⚗): this approach beats standard distillation and lifts accuracy to 85.2%. A convnet teacher (e.g., RegNetY) transfers more than a transformer teacher of similar accuracy.
  • Key training choices:
    • Repeated augmentation (3 repetitions)
    • Mixup, Cutmix, RandAugment, random erasing and stochastic depth
    • Bicubic interpolation of positional embeddings when fine-tuning at higher resolution
  • Transfer and release: transfer results are competitive on CIFAR-10, CIFAR-100, Oxford-102 flowers, Stanford Cars and iNaturalist-18/19. Code is available at github.com/facebookresearch/deit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

254 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 246 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2012.12877 · arXiv 2012.12877

@misc{touvron2020training,
  title = {{Training data-efficient image transformers \& distillation through attention}},
  author = {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve},
  year = {2020},
  doi = {10.48550/arxiv.2012.12877},
  eprint = {2012.12877},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.12877}
}

References

62 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006.00555, 2020. — [arXiv:2006.00555](https://arxiv.org/abs/2006.00555) find
  2. Jie Hu andLi Shen and Gang Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017. — [arXiv:1709.01507](https://arxiv.org/abs/1709.01507) find
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  4. Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. Multigrain: a unified image embedding for classes and instances. arXiv preprint arXiv:1902.05509, 2019. — [arXiv:1902.05509](https://arxiv.org/abs/1902.05509) find
  5. Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) find
  6. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020. find our card
  7. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, 2020. find
  8. Yen-Chun Chen, Linjie Li, Licheng Yu, A. E. Kholy, Faisal Ahmed, Zhe Gan, Y. Cheng, and Jing jing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, 2020. find our card
  9. J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. International Conference on Computer Vision, 2019. find
  10. P. Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. arXiv preprint arXiv:2008.03673, 2020. — [arXiv:2008.03673](https://arxiv.org/abs/2008.03673) find
  11. Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018. — [arXiv:1805.09501](https://arxiv.org/abs/1805.09501) find
  12. Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719, 2019. — [arXiv:1909.13719](https://arxiv.org/abs/1909.13719) find
  13. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  15. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. find
  16. Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019. ICLR 2020. find our card
  17. Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320, 2020. — [arXiv:2004.07320](https://arxiv.org/abs/2004.07320) find
  18. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017. find our card
  19. Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. find
  20. Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. NIPS, 31, 2018. find
  21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, June 2016. find
  22. Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Conference on Computer Vision and Pattern Recognition, 2019. find
  23. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. find
  24. Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. find our card
  25. Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Conference on Computer Vision and Pattern Recognition, 2020. find
  26. Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2018 dataset. arXiv preprint arXiv:1707.06642, 2018. — [arXiv:1707.06642](https://arxiv.org/abs/1707.06642) find
  27. Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2019 dataset. arXiv preprint arXiv:1707.06642, 2019. — [arXiv:1707.06642](https://arxiv.org/abs/1707.06642) find
  28. H. Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Y. Wei. Relation networks for object detection. Conference on Computer Vision and Pattern Recognition, 2018. find
  29. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. find
  30. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013. find
  31. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, CIFAR, 2009. find
  32. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. find
  33. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. find our card
  34. Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. Conference on Computer Vision and Pattern Recognition, 2019. find
  35. Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, A. Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055, 2020. — [arXiv:2006.15055](https://arxiv.org/abs/2006.15055) find
  36. I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 2017. find
  37. Jiasen Lu, Dhruv Batra, D. Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NIPS, 2019. find our card
  38. M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008. find
  39. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026–8037, 2019. find our card
  40. Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. Conference on Computer Vision and Pattern Recognition, 2020. find
  41. B. Recht, Rebecca Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. find
  42. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International journal of Computer Vision, 2015. find our card
  43. Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli, Xuhui Jia, and Ching-Hui Chen. Global self-attention networks for image recognition. arXiv preprint arXiv:2010.03019, 2020. — [arXiv:2010.03019](https://arxiv.org/abs/2010.03019) find
  44. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. find our card
  45. C. Sun, A. Myers, Carl Vondrick, Kevin Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. Conference on Computer Vision and Pattern Recognition, 2019. find our card
  46. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017. find
  47. Christian Szegedy, V. Vanhoucke, S. Ioffe, Jon Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. Conference on Computer Vision and Pattern Recognition, 2016. find
  48. Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, 2019. find our card
  49. Hugo Touvron, Alexandre Sablayrolles, M. Douze, M. Cord, and H. Jégou. Grafit: Learning fine-grained image representations with coarse labels. arXiv preprint arXiv:2011.12982, 2020. — [arXiv:2011.12982](https://arxiv.org/abs/2011.12982) find
  50. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. NIPS, 2019. find
  51. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.08237, 2020. — [arXiv:2003.08237](https://arxiv.org/abs/2003.08237) find
  52. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. find
  53. X. Wang, Ross B. Girshick, A. Gupta, and Kaiming He. Non-local neural networks. Conference on Computer Vision and Pattern Recognition, 2018. find
  54. Longhui Wei, An Xiao, Lingxi Xie, Xin Chen, Xiaopeng Zhang, and Qi Tian. Circumventing outliers of autoaugment with knowledge distillation. European Conference on Computer Vision, 2020. find
  55. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019. ↗
  56. Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677, 2020. — [arXiv:2006.03677](https://arxiv.org/abs/2006.03677) find
  57. Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252, 2019. — [arXiv:1911.04252](https://arxiv.org/abs/1911.04252) find
  58. L. Yuan, F. Tay, G. Li, T. Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. Conference on Computer Vision and Pattern Recognition, 2020. find
  59. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint arXiv:1905.04899, 2019. — [arXiv:1905.04899](https://arxiv.org/abs/1905.04899) find
  60. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. find our card
  61. Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020. — [arXiv:2004.08955](https://arxiv.org/abs/2004.08955) find
  62. Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020. find