Elamur Research
☰
home / papers / 2015 / Microsoft COCO Captions: Data Collection and Evaluation Server

Microsoft COCO Captions: Data Collection and Evaluation Server

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1504.00325

Summary

Comparing image captioning systems was difficult because researchers used different metrics, and even different implementations of the same metric. This paper presents the MS COCO Caption dataset and a public evaluation server that scores every system the same way. Crowd workers on Amazon Mechanical Turk wrote the captions for images from Microsoft COCO. These are Flickr photos chosen to show several objects in everyday scenes. Workers followed fixed rules: at least eight words, no proper names, no unimportant details and no guesses about the past or future. The server accepts generated captions for the validation and test images. The reference captions for the test set stay hidden to reduce overfitting. Before scoring, the server tokenizes the text with the Stanford CoreNLP PTBTokenizer and removes punctuation. It then reports BLEU-1 to BLEU-4, METEOR, ROUGE-L and CIDEr-D. The authors also scored one extra human-written caption per test image, which gives a human baseline. This dataset and protocol became the standard benchmark for neural caption generation.

Key points

  • Two reference sets are provided. MS COCO c5 has five captions for every training, validation and test image. MS COCO c40 has 40 captions for each of 5,000 randomly chosen test images, because metrics agree better with human judgement when there are more references.
  • The authors collected 1,026,459 captions: 413,915 for 82,783 training images, 202,520 for 40,504 validation images and 379,249 for 40,775 test images. They expected the planned MS COCO 2015 release to roughly double this.
  • The paper explains each metric. BLEU counts matching word sequences (precision) and penalizes short captions. ROUGE-L uses the longest common subsequence. METEOR aligns words using exact matches, WordNet synonyms, word stems and paraphrases. CIDEr-D weights word sequences by TF-IDF, so phrases common across the whole dataset count for less.
  • Human captions scored as follows. On c5: BLEU-4 0.217, METEOR 0.252, ROUGE-L 0.484, CIDEr-D 0.854. On c40: BLEU-4 0.471, METEOR 0.335, ROUGE-L 0.626, CIDEr-D 0.910.
  • The authors note that BLEU works well over a whole corpus but poorly for single sentences. They also planned yearly human evaluations to add to the automatic scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

397 papers of our library cite it (of the 140673 whose bibliographies we read); 42 of them have a page here, 355 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1504.00325 · arXiv 1504.00325

@misc{chen2015microsoft,
  title = {{Microsoft COCO Captions: Data Collection and Evaluation Server}},
  author = {Chen, Xinlei and Fang, Hao and Lin, Tsung-Yi and Vedantam, Ramakrishna and Gupta, Saurabh and Dollar, Piotr and Zitnick, C. Lawrence},
  year = {2015},
  doi = {10.48550/arxiv.1504.00325},
  eprint = {1504.00325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1504.00325}
}

References

46 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Barnard and D. Forsyth, “Learning the semantics of words and pictures,” in ICCV, vol. 2, 2001, pp. 408–415. find
  2. K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan, “Matching words and pictures,” JMLR, vol. 3, pp. 1107–1135, 2003. find
  3. V. Lavrenko, R. Manmatha, and J. Jeon, “A model for learning the semantics of pictures,” in NIPS, 2003. find
  4. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Baby talk: Understanding and generating simple image descriptions,” in CVPR, 2011. find
  5. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III, “Midge: Generating image descriptions from computer vision detections,” in EACL, 2012. find
  6. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in ECCV, 2010. find
  7. M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics.” JAIR, vol. 47, pp. 853–899, 2013. find
  8. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, “Collective generation of natural image descriptions,” in ACL, 2012. find
  9. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos, “Corpus-guided sentence generation of natural images,” in EMNLP, 2011. find
  10. A. Gupta, Y. Verma, and C. Jawahar, “Choosing linguistics over vision to describe images.” in AAAI, 2012. find
  11. E. Bruni, G. Boleda, M. Baroni, and N.-K. Tran, “Distributional semantics in technicolor,” in ACL, 2012. find
  12. Y. Feng and M. Lapata, “Automatic caption generation for news images,” TPAMI, vol. 35, no. 4, pp. 797–812, 2013. find
  13. D. Elliott and F. Keller, “Image description using visual dependency representations.” in EMNLP, 2013, pp. 1292–1302. find
  14. A. Karpathy, A. Joulin, and F.-F. Li, “Deep fragment embeddings for bidirectional image sentence mapping,” in NIPS, 2014. find
  15. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, “Improving image-sentence embeddings using large weakly annotated photo collections,” in ECCV, 2014, pp. 529–545. find
  16. R. Mason and E. Charniak, “Nonparametric method for data-driven image captioning,” in ACL, 2014. find
  17. P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi, “Treetalk: Composition and compression of trees for image descriptions,” TACL, vol. 2, pp. 351–362, 2014. find
  18. K. Ramnath, S. Baker, L. Vanderwende, M. El-Saban, S. N. Sinha, A. Kannan, N. Hassan, M. Galley, Y. Yang, D. Ramanan, A. Bergamo, and L. Torresani, “Autocaption: Automatic caption generation for personal photos,” in WACV, 2014. find
  19. A. Lazaridou, E. Bruni, and M. Baroni, “Is this a wampimuk? cross-modal mapping between distributional semantics and the visual world,” in ACL, 2014. find
  20. R. Kiros, R. Salakhutdinov, and R. Zemel, “Multimodal neural language models,” in ICML, 2014. find
  21. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” arXiv preprint arXiv:1410.1090, 2014. — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
  22. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” arXiv preprint arXiv:1411.4555, 2014. find our card
  23. A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” arXiv preprint arXiv:1412.2306, 2014. find our card
  24. R. Kiros, R. Salakhutdinov, and R. S. Zemel, “Unifying visual-semantic embeddings with multimodal neural language models,” arXiv preprint arXiv:1411.2539, 2014. find our card
  25. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” arXiv preprint arXiv:1411.4389, 2014. find our card
  26. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt et al., “From captions to visual concepts and back,” arXiv preprint arXiv:1411.4952, 2014. find
  27. X. Chen and C. L. Zitnick, “Learning a recurrent visual representation for image caption generation,” arXiv preprint arXiv:1411.5654, 2014. — [arXiv:1411.5654](https://arxiv.org/abs/1411.5654) find
  28. R. Lebret, P. O. Pinheiro, and R. Collobert, “Phrase-based image captioning,” arXiv preprint arXiv:1502.03671, 2015. — [arXiv:1502.03671](https://arxiv.org/abs/1502.03671) find
  29. ——, “Simple image description generator via a linear phrase-based approach,” arXiv preprint arXiv:1412.8419, 2014. — [arXiv:1412.8419](https://arxiv.org/abs/1412.8419) find
  30. A. Lazaridou, N. T. Pham, and M. Baroni, “Combining language and vision with a multimodal skip-gram model,” arXiv preprint arXiv:1501.02598, 2015. find
  31. A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” in NIPS, 2012. find
  32. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  33. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR, 2009. find
  34. M. Grubinger, P. Clough, H. Müller, and T. Deselaers, “The iapr tc-12 benchmark: A new evaluation resource for visual information systems,” in LREC Workshop on Language Resources for Content-based Image Retrieval, 2006. find
  35. V. Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs.” in NIPS, 2011. find
  36. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” TACL, vol. 2, pp. 67– 78, 2014. find
  37. J. Chen, P. Kuznetsova, D. Warren, and Y. Choi, “Déjá image-captions: A corpus of expressive image descriptions in repetition,” in NAACL, 2015. find
  38. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in ECCV, 2014. find our card
  39. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in ACL, 2002. find our card
  40. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in ACL Workshop, 2004. find
  41. M. Denkowski and A. Lavie, “Meteor universal: Language specific translation evaluation for any target language,” in EACL Workshop on Statistical Machine Translation, 2014. find our card
  42. R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” arXiv preprint arXiv:1411.5726, 2014. find our card
  43. C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky, “The Stanford CoreNLP natural language processing toolkit,” in Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2014, pp. 55–60. [Online]. Available: http: //ww… find our card
  44. G. A. Miller, “Wordnet: a lexical database for english,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995. find our card
  45. D. Elliott and F. Keller, “Comparing automatic evaluation measures for image description,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, 2014, pp. 452–457. find
  46. C. Callison-Burch, M. Osborne, and P. Koehn, “Re-evaluation the role of bleu in machine translation research.” in EACL, vol. 6, 2006, pp. 249–256. find