Elamur Research
☰
home / papers / 2016 / SPICE: Semantic Propositional Image Caption Evaluation

SPICE: Semantic Propositional Image Caption Evaluation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1607.08822

Summary

Automatic metrics for image captioning, such as Bleu, ROUGE, CIDEr and METEOR, mostly reward shared n-grams. This misleads them in two ways: unrelated captions that share a long phrase score highly, while paraphrases with no common words score poorly. The authors propose SPICE, which compares captions by meaning instead. Candidate and reference captions are first parsed with a dependency parser using Universal Dependency relations. A rule-based system then converts the parses into scene graphs of objects, attributes and relations. The score is an F-score over matching tuples from the candidate and reference graphs. The image itself is ignored, so evaluation is purely textual. The authors tested SPICE against human judgments on MS COCO, Flickr 8K, a composite dataset and PASCAL-50S. It tracks human ratings of machine-generated captions more closely than the n-gram metrics, especially when scores are aggregated per system. Because tuples can be split by category, SPICE can also diagnose specific skills, such as colour recognition or counting. This makes it useful both for ranking captioning models and for analysing their weaknesses.

Key points

  • On MS COCO, SPICE's system-level correlation with human judgments is 0.88, compared with 0.43 for CIDEr and 0.53 for METEOR.
  • At the caption level (Kendall's τ), SPICE gains only modestly:
    • Flickr 8K: 0.45, versus 0.44 for CIDEr, 0.42 for METEOR and 0.73 for inter-human agreement.
    • Composite dataset: 0.39, versus 0.36 for CIDEr and 0.35 for METEOR.
  • On PASCAL-50S with 5 references, SPICE is best at choosing between two model-generated captions (68.2% accuracy versus 66.8% for METEOR). METEOR remains best overall (80.8% versus 78.8%).
  • Object counts are stored as attributes rather than duplicated nodes, so each wrong number counts as one error. An AMR representation with Smatch was tried and performed poorly.
  • An analysis of 2015 COCO Captioning Challenge entries showed that the MSR system surpassed the human F-score on colour attributes, but there was little evidence that any model could count. The code is released at panderson.me/spice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

244 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 234 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1607.08822 · arXiv 1607.08822

@misc{anderson2016spice,
  title = {{SPICE: Semantic Propositional Image Caption Evaluation}},
  author = {Anderson, Peter and Fernando, Basura and Johnson, Mark and Gould, Stephen},
  year = {2016},
  doi = {10.48550/arxiv.1607.08822},
  eprint = {1607.08822},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.08822}
}

References

45 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR. (2015) find our card
  2. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044 (2015) find our card
  3. Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: Data, models and evaluation metrics. JAIR 47 (2013) 853–899 find
  4. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL 2 (2014) 67–78 find
  5. Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV. (2014) find our card
  6. Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., Zitnick, C.L.: Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015) find our card
  7. Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: BabyTalk: Understanding and generating simple image descriptions. PAMI 35(12) (2013) 2891–2903 find
  8. Elliott, D., Keller, F.: Comparing automatic evaluation measures for image description. In: ACL. (2014) 452–457 find
  9. Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., Plank, B.: Automatic description generation from images: A survey of models, datasets, and evaluation measures. JAIR 55 (2016) 409–442 find
  10. Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: ACL. (2002) find our card
  11. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: ACL Workshop. (2004) 25–26 find
  12. Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: Consensus-based image description evaluation. In: CVPR. (2015) find our card
  13. Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: EACL 2014 Workshop on Statistical Machine Translation. (2014) find our card
  14. Giménez, J., Màrquez, L.: Linguistic features for automatic evaluation of heterogenous MT systems. In: ACL Second Workshop on Statistical Machine Translation find
  15. Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Image retrieval using scene graphs. In: CVPR. (2015) find
  16. Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., Manning, C.D.: Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In: EMNLP 4th Workshop on Vision and Language. (2015) find
  17. Wang, C., Xue, N., Pradhan, S.: A transition-based algorithm for AMR parsing. In: HLT-NAACL. (2015) find
  18. Lin, D., Fidler, S., Kong, C., Urtasun, R.: Visual semantic search: Retrieving videos via complex textual queries. In: CVPR. (2014) find
  19. Klein, D., Manning, C.D.: Accurate unlexicalized parsing. In: ACL. (2003) find our card
  20. De Marneffe, M.C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., Manning, C.D.: Universal stanford dependencies: A cross-linguistic typology. In: LREC. Volume 14. (2014) 4585–4592 find our card
  21. Lo, C.k., Tumuluru, A.K., Wu, D.: Fully automatic semantic MT evaluation. In: ACL Seventh Workshop on Statistical Machine Translation. (2012) find
  22. Pradhan, S.S., Ward, W., Hacioglu, K., Martin, J.H., Jurafsky, D.: Shallow semantic parsing using support vector machines. In: HLT-NAACL. (2004) 233–240 find
  23. Ellebracht, L., Ramisa, A., Swaroop, P., Cordero, J., Moreno-Noguer, F., Quattoni., A.: Semantic tuples for evaluation of image sentence generation. EMNLP 4th Workshop on Vision and Language (2015) find
  24. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., Schneider, N.: Abstract meaning representation (AMR) 1.0 specification. EMNLP (2012) 1533–1544 find
  25. Flanigan, J., Thomson, S., Carbonell, J., Dyer, C., Smith, N.A.: A discriminative graph-based parser for the abstract meaning representation. In: ACL. (2014) find
  26. Werling, K., Angeli, G., Manning, C.: Robust subgraph generation improves abstract meaning representation parsing. In: ACL. (2015) find
  27. Cai, S., Knight, K.: Smatch: an evaluation metric for semantic feature structures. In: ACL (2). (2013) 748–752 find
  28. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: CVPR. (2015) 2641–2649 find
  29. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332 (2016) find our card
  30. Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR. (June 2011) find
  31. Hale, J.: A probabilistic Earley parser as a psycholinguistic model. In: NAACL. (2001) 1–8 find our card
  32. Levy, R.: Expectation-based syntactic comprehension. Cognition 106(3) (2008) 1126–1177 find
  33. Stanojević, M., Kamran, A., Koehn, P., Bojar, O.: Results of the WMT15 metrics shared task. In: ACL Tenth Workshop on Statistical Machine Translation. (2015) 256–273 find
  34. Machacek, M., Bojar, O.: Results of the WMT14 metrics shared task. In: ACL Ninth Workshop on Statistical Machine Translation. (2014) 293–301 find our card
  35. Aditya, S., Yang, Y., Baral, C., Fermuller, C., Aloimonos, Y.: From images to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv:1511.03292 (2015) — [arXiv:1511.03292](https://arxiv.org/abs/1511.03292) find
  36. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR. (2015) find our card
  37. Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using Amazon’s Mechanical Turk. In: HLT-NAACL. (2010) 139–147 find
  38. Fang, H., Gupta, S., Iandola, F.N., Srivastava, R., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J.C., Zitnick, C.L., Zweig, G.: From captions to visual concepts and back. In: CVPR. (2015) find
  39. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR. (2015) find our card
  40. Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., Mitchell, M.: Language models for image captioning: The quirks and what works. arXiv preprint arXiv:1505.01809 (2015) find
  41. Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., Yuille, A.L.: Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In: CVPR. (2015) 2533–2541 find
  42. Devlin, J., Gupta, S., Girshick, R.B., Mitchell, M., Zitnick, C.L.: Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467 (2015) — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) find
  43. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 (2014) find
  44. Kolár, M., Hradis, M., Zemcı́k, P.: Technical report: Image captioning with semantically similar images. arXiv preprint arXiv:1506.03995 (2015) — [arXiv:1506.03995](https://arxiv.org/abs/1506.03995) find
  45. Kiros, R., Salakhutdinov, R., Zemel, R.S.: Multimodal neural language models. ICML 14 (2014) 595–603 find