ImageNet Large Scale Visual Recognition Challenge
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1409.0575
Summary
The paper documents ILSVRC, the yearly ImageNet competition that has served as the reference benchmark for large-scale object recognition since 2010, with entries from over fifty institutions. It builds on the PASCAL VOC model of a public dataset plus an annual contest and workshop, but expands it by more than an order of magnitude. Classes grow from 20 to 1000, and images grow from tens of thousands to well over a million. At this scale, small expert teams can no longer do the labeling. The authors therefore built crowdsourcing pipelines on Amazon Mechanical Turk and designed evaluation rules that tolerate imperfect ground truth. Annotations come in two forms: image-level presence/absence labels and object-level bounding boxes. Test labels are kept on a hidden evaluation server. Beyond describing how the dataset was built, the paper does four things:
- traces algorithmic breakthroughs in classification and detection driven by the challenge
- tests whether the results are statistically significant
- analyzes how object properties affect accuracy
- compares top computer vision systems with human annotators
It closes with lessons from five years of running the challenge. The work matters because it gave the field a common large-scale testbed that shaped recognition research.
Key points
- ILSVRC 2010 contained 1,461,406 images, compared with 19,737 in PASCAL VOC 2010. ILSVRC2012 has 1000 classes and 1,431,167 annotated images, while PASCAL VOC 2012 has 20 classes and 21,738 images.
- The source collection, ImageNet, follows the WordNet hierarchy. As of August 2014 it covered 21,841 synsets and 14,197,122 verified images, about 650 per synset.
- The detection track uses 200 categories with roughly 1K instances each. COCO, by comparison, has 91 categories averaging about 27K instances.
- Test annotations have been hidden since the 2010 edition, whose labels were released later. Results are presented at workshops held alternately at ICCV and ECCV.
- The dataset and challenge details are available at image-net.org/challenges/LSVRC/.
Cited by
- Understanding intermediate layers using linear classifier probes
- Flamingo: a Visual Language Model for Few-Shot Learning
- Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
- Experience Grounds Language
- On the Opportunities and Risks of Foundation Models
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model
- Emerging Properties in Self-Supervised Vision Transformers
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- A Simple Framework for Contrastive Learning of Visual Representations
- Long-term Recurrent Convolutional Networks for Visual Recognition and Description
- ImageBind: One Embedding Space To Bind Them All
- Learning Distributed Representations of Sentences from Unlabelled Data
- Classifier-Free Diffusion Guidance
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Averaging Weights Leads to Wider Optima and Better Generalization
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Understanding Black-box Predictions via Influence Functions
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Merging Models with Fisher-Weighted Averaging
- A Survey on Bias and Fairness in Machine Learning
- Representation Learning with Contrastive Predictive Coding
- DINOv2: Learning Robust Visual Features without Supervision
- FiLM: Visual Reasoning with a General Conditioning Layer
- Robust Speech Recognition via Large-Scale Weak Supervision
- Learning multiple visual domains with residual adapters
- LAION-5B: An open large-scale dataset for training next generation image-text models
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Beyond neural scaling laws: beating power law scaling via data pruning
- A Shared Task on Multimodal Machine Translation and Crosslingual Image Description
- A Corpus for Reasoning About Natural Language Grounded in Photographs (NLVR2)
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Training data-efficient image transformers & distillation through attention
- Multimodal Few-Shot Learning with Frozen Language Models
- Show and Tell: A Neural Image Caption Generator
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Modeling Context in Referring Expressions
- mixup: Beyond Empirical Risk Minimization
- Visual7W: Grounded Question Answering in Images
Cite
@misc{russakovsky2014imagenet,
title = {{ImageNet Large Scale Visual Recognition Challenge}},
author = {Russakovsky, Olga and Deng, Jia and Su, Hao and Krause, Jonathan and Satheesh, Sanjeev and Ma, Sean and Huang, Zhiheng and Karpathy, Andrej and Khosla, Aditya and Bernstein, Michael and Berg, Alexander C. and Fei-Fei, Li},
year = {2014},
doi = {10.48550/arxiv.1409.0575},
eprint = {1409.0575},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1409.0575}
}References
- Ross Girshick, Jeff Donahue, Sergio Guadarrama, Trevor Darrell, Jitendra Malik (Girshick et al., 2013, 2014) -- - KAIST department of EE Jun-Cheol Park, Yunhun Jang, Hyungwon Choi, JaeYoung Jun (Chatfield et al., 2014; Jia, 2013) find
- Brno University of Technology Martin Kolář, Michal Hradiš, Pavel Svoboda (Krizhevsky et al., 2012; Mikolov et al., 2013; Jia, 2013) find
- Chinese Academy of Science † , Southeast University ‡ Peihao Huang † , Yongzhen Huang † , Feng Liu ‡ , Zifeng Wu † , Fang Zhao † , Liang Wang † , Tieniu Tan † (Girshick et al., 2014) -- CRIPAC, CASIA Weiqiang Ren, Chong Wang, Yanhua Chen, Kaiqi Huang, Tieniu Tan (Arbeláez et al., 2014) find
- KAIST † , Cldi Inc. ‡ Kyunghyun Paeng † , Donggeun Yoo † , Sunggyun Park † , Jungin Lee ‡ , Anthony S. Paek ‡ , In So Kweon † , Seong Dae Kim † (Krizhevsky et al., 2012; Perronnin et al., 2010) 40.7 find
- The Chinese University of Hong Kong Wanli Ouyang, Ping Luo, Xingyu Zeng, Shi Qiu, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang, Yuanjun Xiong, Chen Qian, Zhenyao Zhu, Ruohui Wang, Chen-Change Loy, Xiaogang Wang, Xiaoou Tang (Ouyang et al., 2014; Ouyang and Wang, 2013) - 40.5 - 43.9 find
- University of Warwick Ben Graham (Graham, 2013; Schmidhuber, 2012) find
- NLPR † , HKUST ‡ Junjie Yan † , Naiyan Wang ‡ , Stan Z. Li † , Dit-Yan Yeung ‡ (Girshick et al., 2014) find
- Fengjun Lv Consulting Fengjun Lv (Krizhevsky et al., 2012; Harel et al., 2007) find
- Google Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Drago Anguelov, Dumitru Erhan, Andrew Rabinovich (Szegedy et al., 2014) Cewu Lu † , Hei Law* † , Hao Chen* ‡ , Qifeng Chen* ∓ , Yao Xiao* † Chi Keung Tang † (Uijlings et al., 2013; - Hong Kong U. of Science and Tech. † , C… find
- Tokyo † , Senthil Purushwalkam † ‡ , Yuichiro Tsuchiya † , Atsushi Harada † (Kanezaki et al., 2014; Girshick et al., 2013) find