Elamur Research
☰
home / papers / 2014 / ImageNet Large Scale Visual Recognition Challenge

ImageNet Large Scale Visual Recognition Challenge

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1409.0575

Summary

The paper documents ILSVRC, the yearly ImageNet competition that has served as the reference benchmark for large-scale object recognition since 2010, with entries from over fifty institutions. It builds on the PASCAL VOC model of a public dataset plus an annual contest and workshop, but expands it by more than an order of magnitude. Classes grow from 20 to 1000, and images grow from tens of thousands to well over a million. At this scale, small expert teams can no longer do the labeling. The authors therefore built crowdsourcing pipelines on Amazon Mechanical Turk and designed evaluation rules that tolerate imperfect ground truth. Annotations come in two forms: image-level presence/absence labels and object-level bounding boxes. Test labels are kept on a hidden evaluation server. Beyond describing how the dataset was built, the paper does four things:

  • traces algorithmic breakthroughs in classification and detection driven by the challenge
  • tests whether the results are statistically significant
  • analyzes how object properties affect accuracy
  • compares top computer vision systems with human annotators

It closes with lessons from five years of running the challenge. The work matters because it gave the field a common large-scale testbed that shaped recognition research.

Key points

  • ILSVRC 2010 contained 1,461,406 images, compared with 19,737 in PASCAL VOC 2010. ILSVRC2012 has 1000 classes and 1,431,167 annotated images, while PASCAL VOC 2012 has 20 classes and 21,738 images.
  • The source collection, ImageNet, follows the WordNet hierarchy. As of August 2014 it covered 21,841 synsets and 14,197,122 verified images, about 650 per synset.
  • The detection track uses 200 categories with roughly 1K instances each. COCO, by comparison, has 91 categories averaging about 27K instances.
  • Test annotations have been hidden since the 2010 edition, whose labels were released later. Results are presented at workshops held alternately at ICCV and ECCV.
  • The dataset and challenge details are available at image-net.org/challenges/LSVRC/.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

762 papers of our library cite it (of the 140673 whose bibliographies we read); 43 of them have a page here, 719 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1409.0575 · arXiv 1409.0575

@misc{russakovsky2014imagenet,
  title = {{ImageNet Large Scale Visual Recognition Challenge}},
  author = {Russakovsky, Olga and Deng, Jia and Su, Hao and Krause, Jonathan and Satheesh, Sanjeev and Ma, Sean and Huang, Zhiheng and Karpathy, Andrej and Khosla, Aditya and Bernstein, Michael and Berg, Alexander C. and Fei-Fei, Li},
  year = {2014},
  doi = {10.48550/arxiv.1409.0575},
  eprint = {1409.0575},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.0575}
}

References

10 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ross Girshick, Jeff Donahue, Sergio Guadarrama, Trevor Darrell, Jitendra Malik (Girshick et al., 2013, 2014) -- - KAIST department of EE Jun-Cheol Park, Yunhun Jang, Hyungwon Choi, JaeYoung Jun (Chatfield et al., 2014; Jia, 2013) find
  2. Brno University of Technology Martin Kolář, Michal Hradiš, Pavel Svoboda (Krizhevsky et al., 2012; Mikolov et al., 2013; Jia, 2013) find
  3. Chinese Academy of Science † , Southeast University ‡ Peihao Huang † , Yongzhen Huang † , Feng Liu ‡ , Zifeng Wu † , Fang Zhao † , Liang Wang † , Tieniu Tan † (Girshick et al., 2014) -- CRIPAC, CASIA Weiqiang Ren, Chong Wang, Yanhua Chen, Kaiqi Huang, Tieniu Tan (Arbeláez et al., 2014) find
  4. KAIST † , Cldi Inc. ‡ Kyunghyun Paeng † , Donggeun Yoo † , Sunggyun Park † , Jungin Lee ‡ , Anthony S. Paek ‡ , In So Kweon † , Seong Dae Kim † (Krizhevsky et al., 2012; Perronnin et al., 2010) 40.7 find
  5. The Chinese University of Hong Kong Wanli Ouyang, Ping Luo, Xingyu Zeng, Shi Qiu, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang, Yuanjun Xiong, Chen Qian, Zhenyao Zhu, Ruohui Wang, Chen-Change Loy, Xiaogang Wang, Xiaoou Tang (Ouyang et al., 2014; Ouyang and Wang, 2013) - 40.5 - 43.9 find
  6. University of Warwick Ben Graham (Graham, 2013; Schmidhuber, 2012) find
  7. NLPR † , HKUST ‡ Junjie Yan † , Naiyan Wang ‡ , Stan Z. Li † , Dit-Yan Yeung ‡ (Girshick et al., 2014) find
  8. Fengjun Lv Consulting Fengjun Lv (Krizhevsky et al., 2012; Harel et al., 2007) find
  9. Google Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Drago Anguelov, Dumitru Erhan, Andrew Rabinovich (Szegedy et al., 2014) Cewu Lu † , Hei Law* † , Hao Chen* ‡ , Qifeng Chen* ∓ , Yao Xiao* † Chi Keung Tang † (Uijlings et al., 2013; - Hong Kong U. of Science and Tech. † , C… find
  10. Tokyo † , Senthil Purushwalkam † ‡ , Yuichiro Tsuchiya † , Atsushi Harada † (Kanezaki et al., 2014; Girshick et al., 2013) find