Elamur Research⌕
☰
home / papers / 2017 / Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Paper · 2017 · Chen Sun, Abhinav Shrivastava, Saurabh Singh et al.

Summary

Since AlexNet, vision models and GPU compute have grown rapidly, yet the largest labeled pre-training set in common use stayed at ImageNet's scale of about 1M images. The authors ask what happens when pre-training data grows by one or two orders of magnitude. They use JFT-300M, a Google-internal collection of 300M images with roughly 375M automatically derived labels over 18,291 categories. About one label in five is wrong. ResNet models are pre-trained on it as multi-label classifiers and then fine-tuned on four downstream problems: classifying images, detecting objects, segmenting scenes semantically and estimating human pose. Random subsets of 10M, 30M and 100M images show how results depend on data volume. Every task improves, and the gains rise roughly linearly with the logarithm of dataset size. Deeper networks benefit more from the extra data. The resulting models set new state-of-the-art results, including 37.4 AP for a single model on COCO detection. The study matters because it moves attention from architecture tuning back to data. It shows that huge, noisy, automatically labeled corpora still produce stronger transferable representations.

Key points

  • JFT-300M: 300M images and 375M labels (about 1.26 per image) across 18,291 categories. The categories form a hierarchy up to 12 levels deep. Labels come from web signals and user feedback, and about 20% are noisy. The distribution is long-tailed: more than 3K classes have under 100 images, and about 2K have under 20.
  • COCO detection: single-model AP rises from 34.3 to 37.4. Model capacity matters: the gain is 1.87% for ResNet-50 but about 3% for ResNet-152.
  • Pascal VOC 2012 segmentation: DeepLab-ASPP-L reaches 73.6 mIOU with ImageNet init, 75.3 with JFT-300M, and 76.5 when JFT training starts from an ImageNet checkpoint.
  • COCO pose estimation (G-RMI Pose setup, test-dev): AP rises from 62.4 with ImageNet init to 64.8 with JFT-300M init.
  • Compute cost: training for only 4 epochs on JFT took 2 months on 50 K-80 GPUs. Hyper-parameters were carried over from 1M-image practice, so the authors believe the reported gains understate what more data can deliver.

Interesting ideas

  • I1. Fill in missing labels from a hierarchy. If an image is tagged with a child class (e.g. "apple"), the training target also includes its ancestors ("fruit"). This is a cheap fix for incomplete multi-label annotation wherever a taxonomy exists, for example product catalogues, geographic place types or lexical resources.
  • I2. Measure data scaling with log-spaced random subsets. Pre-training on nested samples such as 10M, 30M, 100M and 300M, then plotting downstream scores on a log axis, shows whether returns are flattening. The same protocol could tell us whether collecting more Ukrainian text or trade records is still worth it.
  • I3. Match model capacity to data size. The small ResNet-50 gained little from the extra images, so a "more data doesn't help" result may really mean the model was too small. Any scaling experiment should check this before drawing conclusions.
  • I4. Chain pre-training stages. Starting large-scale noisy pre-training from a clean, curated checkpoint (ImageNet, then JFT) beat either source alone in segmentation. This suggests combining a small high-quality corpus with a large weakly labeled one.
  • I5. Treat volume as a counterweight to label noise. Labels that were about 20% wrong still yielded better representations at this scale. This supports using automatically or distantly labeled data when human annotation is out of reach.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Chen Sun, Abhinav Shrivastava, Saurabh Singh and Abhinav Gupta. 2017. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. arXiv:1707.02968.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.02968 · arXiv 1707.02968

@misc{sun2017revisiting,
  title = {{Revisiting Unreasonable Effectiveness of Data in Deep Learning Era}},
  author = {Chen Sun, Abhinav Shrivastava, Saurabh Singh and Abhinav Gupta},
  year = {2017},
  doi = {10.48550/arxiv.1707.02968},
  eprint = {1707.02968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.02968}
}

References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Agrawal, R. B. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV, 2014. find
  2. A. Bergamo and L. Torresani. Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In NIPS. 2010. find
  3. Z. Cao, T. Simon, S. Wei, and Y. Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. arXiv:1611.08050, 2016. find
  4. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915, 2016. find
  5. X. Chen and A. Gupta. Webly supervised learning of convolutional networks. In ICCV, 2015. find
  6. X. Chen, A. Shrivastava, and A. Gupta. Neil: Extracting visual knowledge from web data. In ICCV, 2013. find
  7. F. Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv:1610.02357, 2016. find
  8. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS, 2012. find
  9. S. Divvala, A. Farhadi, and C. Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In CVPR, 2014. find
  10. C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. find
  11. J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv:1605.09782, 2016. find
  12. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 2010. find
  13. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. find
  14. R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv:1311.2524, 2013.
  15. B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In ICCV, 2011. find
  16. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
  17. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS, 2014.
  18. J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR, 2017. find
  19. M. Huh, P. Agrawal, and A. A. Efros. What makes imagenet good for transfer learning? arXiv:1608.08614, 2016. — [arXiv:1608.08614](https://arxiv.org/abs/1608.08614) find
  20. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015. find
  21. H. Izadinia, B. C. Russell, A. Farhadi, M. D. Hoffman, and A. Hertzmann. Deep classifiers from image tags in the wild. In ACM MM, 2015. find
  22. M. Jain, J. C. van Gemert, and C. G. Snoek. What do 15,000 object categories tell us about classifying and localizing actions? In CVPR, 2015. find
  23. A. Joulin, L. van der Maaten, A. Jabri, and N. Vasilache. Learning visual features from large weakly supervised data. arXiv:1511.02251, 2015. — [arXiv:1511.02251](https://arxiv.org/abs/1511.02251) find
  24. J. Krause, B. Sapp, A. Howard, H. Zhou, A. Toshev, T. Duerig, J. Philbin, and F. Li. The unreasonable effectiveness of noisy data for fine-grained recognition. arXiv:1511.06789, 2015. — [arXiv:1511.06789](https://arxiv.org/abs/1511.06789) find
  25. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  26. T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
  27. K. Ni, R. A. Pearce, K. Boakye, B. V. Essen, D. Borth, B. Chen, and E. X. Wang. Large-scale deep learning on the YFCC100M dataset. arXiv:1502.03409, 2015. find
  28. M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. find
  29. G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multi-person pose estimation in the wild. arXiv:1701.01779, 2017. find
  30. F. Pereira, P. Norvig, and A. Halev. The unreasonable effectiveness of data. IEEE Intelligent Systems, 2009. find
  31. L. Pinto, D. Gandhi, Y. Han, Y. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. arXiv:1604.01360, 2016. — [arXiv:1604.01360](https://arxiv.org/abs/1604.01360) find
  32. L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. arXiv:1509.06825, 2015. find
  33. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
  34. M. Rubinstein, A. Joulin, J. Kopf, and C. Liu. Unsupervised joint object discovery and segmentation in internet images. CVPR, 2013. find
  35. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014.
  36. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv:1406.2199, 2014. find
  37. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.
  38. C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv:1602.07261, 2016. — [arXiv:1602.07261](https://arxiv.org/abs/1602.07261) find
  39. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. find
  40. B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li. The new data and new challenges in multimedia research. arXiv:1503.01817, 2015.
  41. A. Torralba and A. Efros. Unbiased look at dataset bias. CVPR, 2011. find
  42. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In NIPS, 2016. find
  43. X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. arXiv:1505.00687, 2015. — [arXiv:1505.00687](https://arxiv.org/abs/1505.00687) find
  44. T. Weyand, I. Kostrikov, and J. Philbin. Planet - photo geolocation with convolutional neural networks. arXiv:1602.05314, 2016. — [arXiv:1602.05314](https://arxiv.org/abs/1602.05314) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2017-revisiting-unreasonable-effectiveness-data-deep/