Elamur Research
☰
home / papers / 2014 / CIDEr: Consensus-based Image Description Evaluation

CIDEr: Consensus-based Image Description Evaluation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1411.5726

Summary

Judging machine-written image captions is hard. Human ratings are slow and costly, and older automatic scores such as BLEU and ROUGE agree poorly with people. This paper changes the goal of evaluation: a caption should resemble how most people would describe the image, rather than get a score for general quality. The paper has three parts. First, a human annotation scheme based on triplets: crowd workers on Amazon Mechanical Turk see one reference sentence and say which of two candidate captions is closer to it. Second, an automatic metric, CIDEr, which compares a candidate with many human references using TF-IDF-weighted n-gram similarity. Third, two datasets with 50 human captions per image, PASCAL-50S and ABSTRACT-50S. Across several kinds of caption pairs, CIDEr matched human consensus judgments better than the existing metrics. Having more references per image helped almost every metric. A hardened version, CIDEr-D, was added to the MS COCO caption evaluation server, and it became a standard benchmark for captioning.

Key points

  • Datasets: PASCAL-50S covers the 1,000 images of the UIUC Pascal Sentence Dataset. ABSTRACT-50S covers 500 images from the Abstract Scenes dataset. Each image has 50 captions, where earlier datasets had at most 5. Partly because of this finding, the MS COCO test set now has 40 references for 5K images.
  • Accuracy at predicting consensus: CIDEr reaches 82% on PASCAL-50S, compared with 90% for humans. On ABSTRACT-50S it reaches 82%, compared with 83% for humans. Its largest gains over other metrics are on pairs of two human captions and on pairs of two machine captions.
  • Ranking five systems: The systems were Midge, Babytalk, Story, Video and Video+. CIDEr's ranking correlates with human judgments at Pearson 0.98. Both put Midge first and Story last.
  • CIDEr-D: This version is harder to game. It drops stemming, adds a Gaussian penalty on length differences (σ = 6), clips n-gram counts and multiplies the score by 10. Its Spearman ρ with the original CIDEr is 0.94.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

693 papers of our library cite it (of the 140673 whose bibliographies we read); 41 of them have a page here, 652 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1411.5726 · arXiv 1411.5726

@misc{vedantam2014cider,
  title = {{CIDEr: Consensus-based Image Description Evaluation}},
  author = {Vedantam, Ramakrishna and Zitnick, C. Lawrence and Parikh, Devi},
  year = {2014},
  doi = {10.48550/arxiv.1411.5726},
  eprint = {1411.5726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1411.5726}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. pages 65–72, 2005. 2, 5, 6 find our card
  2. A. C. Berg, T. L. Berg, H. D. III, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, and K. Yamaguchi. Understanding and predicting importance in images. In CVPR. IEEE, 2012. 2 find
  3. R. Bogacz, E. Brown, J. Moehlis, P. Holmes, and J. D. Cohen. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol Rev, 113(4):700–765, Oct. 2006. 1, 4 find
  4. C. Callison-burch and M. Osborne. Re-evaluating the role of bleu in machine translation research. In In EACL, pages 249–256, 2006. 1, 2 find
  5. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. ArXiv e-prints, Apr. 2015. 2, 8 find our card
  6. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. CoRR, abs/1411.5654, 2014. 2, 3 — [arXiv:1411.5654](https://arxiv.org/abs/1411.5654) find
  7. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. 1 find
  8. M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014. 6 find our card
  9. P. K. Dokania, A. Behl, C. V. Jawahar, and P. M. Kumar. Learning to rank using high-order information. ECCV, 2014. 1 find
  10. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. CoRR, abs/1411.4389, 2014. 2, 3 find our card
  11. D. Elliott and F. Keller. Image description using visual dependency representations. In EMNLP, pages 1292–1302. ACL, 2013. 1, 2, 3 find
  12. D. Elliott and F. Keller. Comparing automatic evaluation measures for image description. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 452–457, Baltimore, Maryland, June 2014. Association for Computational Linguistics. 3, 5… find
  13. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html.1 ↗
  14. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In Proceedings of the 11th European Conference on Computer Vision: Part IV, ECCV’10, 2010. 2, 5, 6, 7, 12, 16 find
  15. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627–1645, 2010. 1 find
  16. A. Gupta and L. S. Davis. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In D. A. Forsyth, P. H. S. Torr, and A. Zisserman, editors, ECCV (1), volume 5302 of Lecture Notes in Computer Science, pages 16–29. Springer, 2008. 2 find
  17. A. Gupta, Y. Verma, and C. Jawahar. Choosing linguistics over vision to describe images. 2012. 2 find
  18. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res. (JAIR), 47:853–899, 2013. 1, 2, 3, 5, 7 find
  19. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. CoRR, abs/1412.2306, 2014. 2, 3 find our card
  20. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. CoRR, 2014. 2 find
  21. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539, 2014. 2, 3 find our card
  22. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating image descriptions. In Proceedings of the 24th CVPR, 2011. 1, 2, 5, 6, 7, 11, 12, 16 find
  23. C. H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by betweenclass attribute transfer. In In CVPR, 2009. 1, 2 find
  24. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, CoNLL ’11, pages 220– 228, Stroudsburg, PA, USA, 2011. Association for Computational Linguistic… find
  25. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 3, 4 find our card
  26. S. Maji, L. Bourdev, and J. Malik. Action recognition from a distributed representation of pose and appearance. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011. 1 find
  27. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. CoRR, abs/1410.1090, 2014. 2, 3 — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
  28. D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision, volume 2, pages 416–423, July 2001. 1 find
  29. M. Mitchell, X. Han, and J. Hayes. Midge: Generating descriptions of images. In Proceedings of the Seventh International Natural Language Generation Conference, INLG ’12, pages 131–133, Stroudsburg, PA, USA, 2012. Association for Computational Linguistics. 1, 2, 5, 6, 7, 12, 16 find
  30. H. Mller, P. Clough, T. Deselaers, and B. Caputo. Image-CLEF: Experimental Evaluation in Visual Information Retrieval. Springer Publishing Company, Incorporated, 1st edition, 2010. 3 find
  31. A. Nenkova and R. J. Passonneau. Evaluating content selection in summarization: The pyramid method. In HLT-NAACL, pages 145–152, 2004. 3 find
  32. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS), 2011. 2, 3, 4, 11 find
  33. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA, 2002. Association for Computational Linguistics. 1, 2,… find our card
  34. D. Parikh and K. Grauman. Relative Attributes. In ICCV, 2011. 2 find
  35. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, Stroudsburg, PA, USA, 2010. Association for Computationa… find
  36. S. Robertson. Understanding inverse document frequency: On theoretical arguments for idf. Journal of Documentation, 60:2004, 2004. 4 find
  37. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele. Translating video content to natural language descriptions. In IEEE International Conference on Computer Vision (ICCV), December 2013. 1, 2, 5, 6, 8, 12, 16 find
  38. M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. 2011. 2 find
  39. D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision, 2002. 1 find
  40. A. Sorokin and D. Forsyth. Utility data annotation with amazon mechanical turk. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08. IEEE Computer Society Conference on, pages 1–8, June 2008. 1 find
  41. O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. T. Kalai. Adaptively learning the crowd kernel. In In ICML11, 2011. 1 find
  42. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. CoRR, abs/1411.4555, 2014. 2, 3 find our card
  43. Y. Yang, C. L. Teo, H. D. III, and Y. Aloimonos. Corpus-guided sentence generation of natural images. In EMNLP. ACL, 2011. 1, 2 find
  44. M. Yatskar, M. Galley, L. Vanderwende, and L. Zettlemoyer. See no evil, say no evil: Description generation from densely labeled images. In Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), page 110120, Dublin, Ireland, August 2014. Association for Computa… find
  45. C. yew Lin. Rouge: a package for automatic evaluation of summaries. pages 25–26, 2004. 1, 3, 5 find
  46. C. L. Zitnick and D. Parikh. Bringing semantics into focus using visual abstraction. In CVPR, 2013. 2, 3, 4 find