Elamur Research
☰
home / papers / 2014 / Show and Tell: A Neural Image Caption Generator

Show and Tell: A Neural Image Caption Generator

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1411.4555

Summary

The paper addresses automatic image captioning: writing a fluent English sentence that names the objects in a picture and also describes their attributes, how they relate and what they are doing. Earlier systems either combined object detectors with templates and hand-written rules, or retrieved existing captions. The authors instead propose NIC (Neural Image Caption), a single network trained end to end with stochastic gradient descent. It borrows the encoder–decoder approach from neural machine translation but replaces the source-language RNN with a deep CNN. That CNN is pretrained for image classification and uses the batch-normalized architecture that topped ILSVRC 2014. The image vector it produces goes into an LSTM, which generates the caption one word at a time using learned word embeddings. Training maximizes the log-likelihood of the reference captions. On Pascal, Flickr8k, Flickr30k, SBU and MSCOCO, NIC clearly beats earlier caption generators on BLEU and does well on retrieval-style ranking, though human raters still prefer human-written captions. NIC showed that a general sequence model can learn to describe images directly from image–caption pairs, and it became a standard baseline in vision–language research.

Key points

  • BLEU-1: Pascal rises from 25 (previous best) to 59, against about 69 for humans. Flickr30k goes from 56 to 66 and SBU from 19 to 28. On MSCOCO, NIC reaches a state-of-the-art BLEU-4 of 27.7.
  • Human ratings (average scores): COCO-1k 2.72, Pascal 2.45 and Flickr8k 2.37. A reference system scored 2.08 and ground-truth captions 3.89. This gap suggests BLEU makes NIC look closer to human quality than it is.
  • Ranking on Flickr8k: NIC beats DeFrag, m-RNN and MNLM, with R@1 of 20 for annotation and 19 for search and median ranks of 6 and 5. On Flickr30k it falls behind MNLM, which was trained with a ranking-specific loss.
  • Novelty: the top-scoring caption copies a training sentence 80% of the time. Among the 15 best candidates, roughly half are new sentences with similar BLEU scores.
  • Word embeddings: the learned embeddings group related words together, for example horse near pony, donkey and mule, and car near van, cab and jeep.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

470 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 432 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1411.4555 · arXiv 1411.4555

@misc{vinyals2014show,
  title = {{Show and Tell: A Neural Image Caption Generator}},
  author = {Vinyals, Oriol and Toshev, Alexander and Bengio, Samy and Erhan, Dumitru},
  year = {2014},
  doi = {10.48550/arxiv.1411.4555},
  eprint = {1411.4555},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1411.4555}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Aker and R. Gaizauskas. Generating image descriptions using dependency relational patterns. In ACL, 2010. find
  2. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014. find our card
  3. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014. find
  4. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014. find
  5. D. Elliott and F. Keller. Image description using visual dependency representations. In EMNLP, 2013. find
  6. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, 2010. find
  7. R. Gerber and H.-H. Nagel. Knowledge representation for the generation of quantified natural language descriptions of vehicle traffic in image sequences. In ICIP. IEEE, 1996. find
  8. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In ECCV, 2014. find
  9. A. Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850, 2013. find
  10. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8), 1997. find
  11. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47, 2013. find
  12. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In arXiv:1502.03167, 2015. find
  13. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. NIPS, 2014. find
  14. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In arXiv:1411.2539, 2014. find our card
  15. R. Kiros and R. Z. R. Salakhutdinov. Multimodal neural language models. In NIPS Deep Learning Workshop, 2013. find
  16. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR, 2011. find
  17. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In ACL, 2012. find
  18. P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi. Treetalk: Composition and compression of trees for image descriptions. ACL, 2(10), 2014. find
  19. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In Conference on Computational Natural Language Learning, 2011. find
  20. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. arXiv:1405.0312, 2014. find our card
  21. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille. Explain images with multimodal recurrent neural networks. In arXiv:1410.1090, 2014. — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
  22. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In ICLR, 2013. find
  23. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. C. Berg, K. Yamaguchi, T. L. Berg, K. Stratos, and H. D. III. Midge: Generating image descriptions from computer vision detections. In EACL, 2012. find
  24. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011. find
  25. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL, 2002. find our card
  26. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon’s mechanical turk. In NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 139– 147, 2010. find
  27. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, 2014. find our card
  28. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. find
  29. R. Socher, A. Karpathy, Q. V. Le, C. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. In ACL, 2014. find our card
  30. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  31. R. Vedantam, C. L. Zitnick, and D. Parikh. CIDEr: Consensus-based image description evaluation. In arXiv:1411.5726, 2015. find our card
  32. B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. I2t: Image parsing to text description. Proceedings of the IEEE, 98(8), 2010. find
  33. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In ACL, 2014. find
  34. W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. In arXiv:1409.2329, 2014. find our card