Elamur Research
☰
home / papers / 2014 / Microsoft COCO: Common Objects in Context

Microsoft COCO: Common Objects in Context

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1405.0312

Summary

Before MS COCO, most object recognition benchmarks rewarded systems that did well on clean, centered "iconic" photos, and they measured localization only with coarse bounding boxes. Microsoft Common Objects in COntext (MS COCO) was built to move the field toward full scene understanding. It targets three things: recognizing objects that are seen from unusual angles, partly hidden, small or in clutter; reasoning about which objects appear together; and localizing each object exactly at the pixel level. The authors collected photos of everyday scenes by searching the web for pairs of objects and for scene types. Crowd workers on Amazon Mechanical Turk then labeled the images in stages: first which categories are present (in a hierarchy), then where each instance is, and finally a verified outline for each instance. The finished resource has 91 everyday "thing" categories and about 2.5 million labeled instances in 328k images. Compared with PASCAL VOC, ImageNet and SUN, it has many more instances per image and per category. Baseline experiments with a Deformable Parts Model show that detectors trained on COCO handle cluttered real scenes better. COCO became a standard benchmark for detection and instance segmentation.

Key points

  • 91 categories were chosen from 272 candidates and grouped into 11 super-categories. For 82 of them there are more than 5,000 labeled instances. 11 of the 91 were left out of the 2014 release.
  • On average there are 7.7 labeled instances per image, compared with 3.0 for ImageNet and 2.3 for PASCAL VOC. SUN has more labels per image but far fewer instances in total.
  • Only countable "thing" classes (person, chair, car) are labeled. Background "stuff" such as sky or grass is left out because the goal is precise localization of individual instances.
  • Pairing object queries with scene queries (e.g., kitchen, street, beach) produced images with context and non-iconic views.
  • Baselines use a Deformable Parts Model for both bounding-box and segmentation detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1981 papers of our library cite it (of the 140673 whose bibliographies we read); 102 of them have a page here, the first 100 below, 1879 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1405.0312 · arXiv 1405.0312

@misc{lin2014microsoft,
  title = {{Microsoft COCO: Common Objects in Context}},
  author = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Bourdev, Lubomir and Girshick, Ross and Hays, James and Perona, Pietro and Ramanan, Deva and Zitnick, C. Lawrence and Dollár, Piotr},
  year = {2014},
  doi = {10.48550/arxiv.1405.0312},
  eprint = {1405.0312},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1405.0312}
}

References

51 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR, 2009. find
  2. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL visual object classes (VOC) challenge,” IJCV, vol. 88, no. 2, pp. 303–338, Jun. 2010. find
  3. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “SUN database: Large-scale scene recognition from abbey to zoo,” in CVPR, 2010. find
  4. P. Dollár, C. Wojek, B. Schiele, and P. Perona, “Pedestrian detection: An evaluation of the state of the art,” PAMI, vol. 34, 2012. find
  5. A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” in NIPS, 2012. find
  6. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in CVPR, 2014. find our card
  7. P. Sermanet, D. Eigen, S. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “OverFeat: Integrated recognition, localization and detection using convolutional networks,” in ICLR, April 2014. find
  8. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in CVPR, 2009. find
  9. G. Patterson and J. Hays, “SUN attribute database: Discovering, annotating, and recognizing scene attributes,” in CVPR, 2012. find
  10. L. Bourdev and J. Malik, “Poselets: Body part detectors trained using 3D human pose annotations,” in ICCV, 2009. find
  11. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in ECCV, 2012. find
  12. S. Palmer, E. Rosch, and P. Chase, “Canonical perspective and the perception of objects,” Attention and performance IX, vol. 1, p. 4, 1981. find
  13. . Hoiem, D, Y. Chodpathumwan, and Q. Dai, “Diagnosing error in object detectors,” in ECCV, 2012. find
  14. G. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A high-definition ground truth database,” PRL, vol. 30, no. 2, pp. 88–97, 2009. find
  15. B. Russell, A. Torralba, K. Murphy, and W. Freeman, “LabelMe: a database and web-based tool for image annotation,” IJCV, vol. 77, no. 1-3, pp. 157–173, 2008. find
  16. S. Bell, P. Upchurch, N. Snavely, and K. Bala, “OpenSurfaces: A richly annotated catalog of surface appearance,” SIGGRAPH, vol. 32, no. 4, 2013. find
  17. V. Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs.” in NIPS, 2011. find
  18. J. Deng, O. Russakovsky, J. Krause, M. Bernstein, A. Berg, and L. Fei-Fei, “Scalable multi-label annotation,” in CHI, 2014. find
  19. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in ECCV, 2014. find
  20. D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” IJCV, vol. 47, no. 1-3, pp. 7–42, 2002. find
  21. S. Baker, D. Scharstein, J. Lewis, S. Roth, M. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” IJCV, vol. 92, no. 1, pp. 1–31, 2011. find
  22. L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in CVPR Workshop of Generative Model Based Vision (WGMBV), 2004. find
  23. G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category dataset,” California Institute of Technology, Tech. Rep. 7694, 2007. find
  24. N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in CVPR, 2005. find
  25. Y. Lecun and C. Cortes, “The MNIST database of handwritten digits,” 1998. [Online]. Available: http://yann.lecun.com/exdb/mnist/ ↗
  26. S. A. Nene, S. K. Nayar, and H. Murase, “Columbia object image library (coil-20),” Columbia Universty, Tech. Rep., 1996. find
  27. A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Computer Science Department, University of Toronto, Tech. Rep, 2009. find
  28. A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” PAMI, vol. 30, no. 11, pp. 1958–1970, 2008. find
  29. V. Ordonez, J. Deng, Y. Choi, A. Berg, and T. Berg, “From large scale image categorization to entry-level categories,” in ICCV, 2013. find
  30. C. Fellbaum, WordNet: An electronic lexical database. Blackwell Books, 1998. find
  31. P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-UCSD Birds 200,” Caltech, Tech. Rep. CNS-TR-201, 2010. find
  32. E. Hjelmås and B. Low, “Face detection: A survey,” CVIU, vol. 83, no. 3, pp. 236–274, 2001. find
  33. G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild,” University of Massachusetts, Amherst, Tech. Rep. 07-49, October 2007. find
  34. O. Russakovsky, J. Deng, Z. Huang, A. Berg, and L. Fei-Fei, “Detecting avocados to zucchinis: what have we done, and where are we going?” in ICCV, 2013. find
  35. J. Shotton, J. Winn, C. Rother, and A. Criminisi, “TextonBoost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context,” IJCV, vol. 81, no. 1, pp. 2–23, 2009. find
  36. S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, “A comparison and evaluation of multi-view stereo reconstruction algorithms,” in CVPR, 2006. find
  37. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” PAMI, vol. 33, no. 5, pp. 898–916, 2011. find
  38. C. Lampert, H. Nickisch, and S. Harmeling, “Learning to detect unseen object classes by between-class attribute transfer,” in CVPR, 2009. find
  39. G. Heitz and D. Koller, “Learning spatial context: Using stuff to find things,” in ECCV, 2008. find
  40. R. Sitton, Spelling Sourcebook. Egger Publishing, 1996. find
  41. T. Berg and A. Berg, “Finding iconic images,” in CVPR, 2009. find
  42. A. Torralba and A. Efros, “Unbiased look at dataset bias,” in CVPR, 2011. find
  43. M. Douze, H. Jégou, H. Sandhawalia, L. Amsaleg, and C. Schmid, “Evaluation of gist descriptors for web-scale image search,” in CIVR, 2009. find
  44. P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” PAMI, vol. 32, no. 9, pp. 1627–1645, 2010. find
  45. R. Girshick, P. Felzenszwalb, and D. McAllester, “Discriminatively trained deformable part models, release 5,” PAMI, 2012. find
  46. X. Zhu, C. Vondrick, D. Ramanan, and C. Fowlkes, “Do we need more training data or better models for object detection?” in BMVC, 2012. find
  47. T. Brox, L. Bourdev, S. Maji, and J. Malik, “Object segmentation by alignment of poselet activations to image contours,” in CVPR, 2011. find
  48. Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes, “Layered object models for image segmentation,” PAMI, vol. 34, no. 9, pp. 1731–1743, 2012. find
  49. D. Ramanan, “Using segmentation to verify object hypotheses,” in CVPR, 2007. find
  50. Q. Dai and D. Hoiem, “Learning to localize detected objects,” in CVPR, 2012. find
  51. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, “Collecting image annotations using Amazon’s Mechanical Turk,” in NAACL Workshop, 2010. Fig. 11: Icons of 91 categories in the MS COCO dataset grouped by 11 super-categories. We use these icons in our annotation pipeline to help workers quickly… find