Elamur Research⌕
☰
home / papers / 2014 / From Captions to Visual Concepts and Back

From Captions to Visual Concepts and Back

Paper · 2014 · Hao Fang, Saurabh Gupta, Forrest Iandola et al.

Summary

The paper tackles automatic image captioning. Its central idea is that every component of the system can be learned from caption-annotated images, with no separately hand-labeled object, attribute or relation data. The pipeline has three stages:

  • Word detection. Detectors for frequent caption words (nouns, verbs and adjectives alike) are trained with weakly supervised Multiple Instance Learning (MIL) on CNN features of image regions.
  • Sentence generation. The detector scores condition a maximum-entropy language model trained on more than 400,000 captions. A search then looks for high-likelihood sentences that use the detected words.
  • Re-ranking. Candidate sentences are re-ranked with MERT-tuned sentence features, including a new Deep Multimodal Similarity Model (DMSM) that embeds images and text in a shared vector space.

On Microsoft COCO, the system was state-of-the-art on all 14 official metrics when the paper was written. It matched or beat human captions on 12 of them. Human judges rated its captions as good as or better than human-written ones about a third of the time. The work shows that captions alone are enough supervision for detectors, language statistics and global image–text similarity.

Key points

  • The vocabulary is the 1000 most frequent caption words, covering over 92% of word occurrences. Detectors use a noisy-OR MIL layer on top of a fully convolutional AlexNet or VGG, which produces a 12×12 response map per image.
  • Training uses the Microsoft COCO train split: 82,783 images with 5 captions each.
  • On the COCO evaluation server (40,775 test images), BLEU-4 reaches 29.1% versus 21.7% for humans. CIDEr is 0.912 versus 0.854 for humans, and this was the only system to beat humans on CIDEr.
  • On the authors' held-out set, the best variant (VGG+Score+DMSM+ft) gets 25.7% BLEU (humans 19.3%) and is judged same-or-better than humans 34% of the time. Replacing MIL with whole-image classification lowers BLEU to 21.9%.
  • On the PASCAL sentence dataset (847 images), the system strongly beats Midge: 17.6% vs 2.0% BLEU and 19.2% vs 9.2% METEOR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓University of WashingtonSeattle · United States402 works · 212,763 citations🏢MetaMenlo Park · United States214 works · 241,492 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland10 works · 8,825 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick and Geoffrey Zweig. 2014. From Captions to Visual Concepts and Back. arXiv:1411.4952.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1411.4952 · arXiv 1411.4952

@misc{fang2014captions,
  title = {{From Captions to Visual Concepts and Back}},
  author = {Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick and Geoffrey Zweig},
  year = {2014},
  doi = {10.48550/arxiv.1411.4952},
  eprint = {1411.4952},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1411.4952}
}

References

51 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005. 2, 6
  2. A. L. Berger, S. A. D. Pietra, and V. J. D. Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 1996. 2, 4
  3. A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka Jr, and T. M. Mitchell. Toward an architecture for never-ending language learning. In AAAI, 2010. 2 find
  4. X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2, 6, 7
  5. X. Chen, A. Shrivastava, and A. Gupta. Neil: Extracting visual knowledge from web data. In ICCV, 2013. 1 find
  6. X. Chen and C. L. Zitnick. Mind’s eye: A recurrent visual representation for image caption generation. CVPR, 2015. 2 find
  7. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 2, 3 find
  8. S. Divvala, A. Farhadi, and C. Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In CVPR, 2014. 1 find
  9. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. CVPR, 2015. 2, 3
  10. D. Elliott and F. Keller. Comparing automatic evaluation measures for image description. In ACL, 2014. 6 find
  11. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL visual object classes (VOC) challenge. IJCV, 88(2):303–338, June 2010. 2, 6 find
  12. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, 2010. 2 find
  13. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 3
  14. G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. Using k-poselets for detecting people and localizing their keypoints. In CVPR, 2014. 4 find
  15. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47:853–899, 2013. 2 find
  16. P. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. Learning deep structured semantic models for web search using clickthrough data. In CIKM, 2013. 5 find
  17. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 3
  18. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. CVPR, 2015. 2
  19. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. arXiv preprint arXiv:1406.5679, 2014. 2 find
  20. R. Kiros, R. Zemel, and R. Salakhutdinov. Multimodal neural language models. In NIPS Deep Learning Workshop, 2013. 2, 3 find
  21. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012. 2, 3, 6, 7
  22. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR, 2011. 1, 2, 6, 8 find
  23. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In ACL, 2012. 2 find
  24. R. Lau, R. Rosenfeld, and S. Roukos. Trigger-based language models: A maximum entropy approach. In ICASSP, 1993. 4 find
  25. R. Lebret, P. O. Pinheiro, and R. Collobert. Phrase-based image captioning. arXiv preprint arXiv:1502.03671, 2015. 2, 3 — [arXiv:1502.03671](https://arxiv.org/abs/1502.03671) find
  26. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In CoNLL, 2011. 2 find
  27. C.-Y. Lin and F. J. Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL ’04, Stroudsburg, PA, USA, 2004. Association for Computational Linguis…
  28. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, 6
  29. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. arXiv preprint arXiv:1410.1090, 2014. 2, 3 find
  30. O. Maron and T. Lozano-Pérez. A framework for multiple-instance learning. NIPS, 1998. 2, 3 find
  31. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky. Strategies for training large scale neural network language models. In ASRU, 2011. 4 find
  32. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III. Midge: Generating image descriptions from computer vision detections. In EACL, 2012. 2, 6, 8 find
  33. A. Mnih and G. Hinton. Three new graphical models for statistical language modelling. In ICML, 2007. 4 find
  34. A. Mnih and Y. W. Teh. A fast and simple algorithm for training neural probabilistic language models. In ICML, 2012. 4
  35. F. J. Och. Minimum error rate training in statistical machine translation. In ACL, 2003. 2, 5 find
  36. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011. 2 find
  37. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. 2, 6
  38. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using Amazon’s mechanical turk. In NAACL HLT Workshop Creating Speech and Language Data with Amazon’s Mechanical Turk, 2010. 2, 6, 8 — Collecting Image Annotations Using Amazon's Mechanical Turk
  39. A. Ratnaparkhi. Trainable methods for surface natural language generation. In NAACL, 2000. 4 find
  40. A. Ratnaparkhi. Trainable approaches to surface natural language generation and their application to conversational dialog systems. Computer Speech & Language, 16(3):435–455, 2002. 2, 3 find
  41. Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014. 5 find
  42. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 2, 3, 6, 7
  43. R. Socher, Q. Le, C. Manning, and A. Ng. Grounded compositional semantics for finding and describing images with sentences. In NIPS Deep Learning Workshop, 2013. 2
  44. R. Vedantam, C. L. Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. CoRR, abs/1411.5726, 2014. 6, 8
  45. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. CVPR, 2015. 2
  46. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015. 2, 3
  47. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos. Corpus-guided sentence generation of natural images. In EMNLP, 2011. 1, 2 find
  48. B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. I2T: Image parsing to text description. Proceedings of the IEEE, 98(8):1485–1508, 2010. 2 find
  49. C. Zhang, J. C. Platt, and P. A. Viola. Multiple instance boosting for object detection. In NIPS, 2005. 2, 3 find
  50. C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014. 8 find
  51. C. L. Zitnick and D. Parikh. Bringing semantics into focus using visual abstraction. In CVPR, 2013. 1 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fang-2014-captions-visual-concepts-back/