Elamur Research
☰
home / papers / 2014 / Deep Visual-Semantic Alignments for Generating Image Descriptions

Deep Visual-Semantic Alignments for Generating Image Descriptions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1412.2306

Summary

The paper tackles automatic generation of natural-language descriptions for whole images and for individual image regions, without fixed sentence templates or a closed set of categories. Caption datasets such as Flickr8K, Flickr30K and MSCOCO mention many objects but never say where they appear, so the authors treat captions as weak supervision. The method has two stages. First, an alignment model maps image regions (found with an RCNN detector) and words (encoded by a bidirectional RNN over word2vec vectors) into a shared multimodal space. A structured ranking objective then learns which contiguous phrase belongs to which region. Second, a Multimodal RNN, which receives the image features as a bias at its first step, learns to generate text. It is trained either on full images or on the region–phrase pairs found in the first stage. The alignment model set a new state of the art for image–sentence retrieval on all three benchmarks, and the generator clearly beat retrieval baselines. This work was an early and influential demonstration of neural models that link free-form language to visual content.

Key points

  • Each image is represented by 20 vectors: the top 19 RCNN detections plus the whole frame. The CNN is pre-trained on ImageNet and fine-tuned on the 200 ImageNet Detection classes. The embedding size h ranges from 1000 to 1600.
  • The BRNN sentence encoder uses fixed 300-dimensional word2vec embeddings and ReLU activations. It replaces the dependency-tree fragments used in earlier work by Karpathy et al.
  • The authors built a new region-level test set on Amazon Mechanical Turk with 9,000 short phrases for 200 MSCOCO test images. Nine annotators worked on each image, and the phrases average 2.3 words.
  • On region descriptions, the region-level RNN scored 35.2 BLEU-1, compared with 14.2 for the full-frame model and 22.9 for the nearest-neighbor baseline. It also scored higher on CIDEr (61.6 vs. 20.3), METEOR and ROUGE.
  • The authors name these limitations: the model sees only one fixed-resolution input, the image enters only through an additive bias, and the two models are not trained end-to-end. Code, data and annotations are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

379 papers of our library cite it (of the 140673 whose bibliographies we read); 41 of them have a page here, 338 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1412.2306 · arXiv 1412.2306

@misc{karpathy2014deep,
  title = {{Deep Visual-Semantic Alignments for Generating Image Descriptions}},
  author = {Karpathy, Andrej and Fei-Fei, Li},
  year = {2014},
  doi = {10.48550/arxiv.1412.2306},
  eprint = {1412.2306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.2306}
}

References

60 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy, D. Salvi, et al. Video in sentences out. arXiv preprint arXiv:1204.2742, 2012. — [arXiv:1204.2742](https://arxiv.org/abs/1204.2742) find
  2. K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan. Matching words and pictures. JMLR, 2003. find
  3. Y. Bengio, H. Schwenk, J.-S. Senécal, F. Morin, and J.-L. Gauvain. Neural probabilistic language models. In Innovations in Machine Learning. Springer, 2006. find
  4. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. find our card
  5. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. CoRR, abs/1411.5654, 2014. — [arXiv:1411.5654](https://arxiv.org/abs/1411.5654) find
  6. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. find
  7. M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014. find our card
  8. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411.4389, 2014. find our card
  9. D. Elliott and F. Keller. Image description using visual dependency representations. In EMNLP, pages 1292–1302, 2013. find
  10. J. L. Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990. find
  11. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303– 338, June 2010. find
  12. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014. find
  13. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV. 2010. find
  14. L. Fei-Fei, A. Iyer, C. Koch, and P. Perona. What do we perceive in a glance of a real-world scene? Journal of vision, 7(1):10, 2007. find
  15. S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth a thousand pixels. In CVPR, 2013. find
  16. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013. find
  17. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. find our card
  18. S. Gould, R. Fulton, and D. Koller. Decomposing a scene into geometric and semantically consistent regions. In Computer Vision, 2009 IEEE 12th International Conference on, pages 1–8. IEEE, 2009. find
  19. A. Gupta and P. Mannem. From image annotation to image description. In Neural information processing. Springer, 2012. find
  20. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  21. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: data, models and evaluation metrics. Journal of Artificial Intelligence Research, 2013. find
  22. R. JeffreyPennington and C. Manning. Glove: Global vectors for word representation. 2014. find
  23. Y. Jia, M. Salzmann, and T. Darrell. Learning cross-modality similarity for multinomial data. In ICCV, 2011. find
  24. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. arXiv preprint arXiv:1406.5679, 2014. — [arXiv:1406.5679](https://arxiv.org/abs/1406.5679) find
  25. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. find our card
  26. R. Kiros, R. S. Zemel, and R. Salakhutdinov. Multimodal neural language models. ICML, 2014. find
  27. C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. find
  28. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. find
  29. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR, 2011. find
  30. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In ACL, 2012. find
  31. P. Kuznetsova, V. Ordonez, T. L. Berg, U. C. Hill, and Y. Choi. Treetalk: Composition and compression of trees for image descriptions. Transactions of the Association for Computational Linguistics, 2(10):351–362, 2014. find
  32. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  33. L.-J. Li and L. Fei-Fei. What, where and who? classifying events by scene and object recognition. In ICCV, 2007. find
  34. L.-J. Li, R. Socher, and L. Fei-Fei. Towards total scene understanding: Classification, annotation and segmentation in an automatic framework. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 2036–2043. IEEE, 2009. find
  35. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In CoNLL, 2011. find
  36. D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual semantic search: Retrieving videos via complex textual queries. 2014. find
  37. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2014. find our card
  38. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. arXiv preprint arXiv:1410.1090, 2014. — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
  39. C. Matuszek*, N. FitzGerald*, L. Zettlemoyer, L. Bo, and D. Fox. A Joint Model of Language and Perception for Grounded Attribute Learning. In Proc. of the 2012 International Conference on Machine Learning, Edinburgh, Scotland, June 2012. find
  40. T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  41. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013. find
  42. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé, III. Midge: Generating image descriptions from computer vision detections. In EACL, 2012. find
  43. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011. find
  44. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002. find our card
  45. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge, 2014. find our card
  46. M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 1997. find
  47. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. find our card
  48. R. Socher and L. Fei-Fei. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In CVPR, 2010. find
  49. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. TACL, 2014. find our card
  50. I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. In ICML, 2011. find
  51. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  52. T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude., 2012. find
  53. R. Vedantam, C. L. Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. CoRR, abs/1411.5726, 2014. find our card
  54. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014. find our card
  55. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos. Corpus-guided sentence generation of natural images. In EMNLP, 2011. find
  56. B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. I2t: Image parsing to text description. Proceedings of the IEEE, 98(8):1485–1508, 2010. find
  57. M. Yatskar, L. Vanderwende, and L. Zettlemoyer. See no evil, say no evil: Description generation from densely labeled images. Lexical and Computational Semantics, 2014. find
  58. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014. find
  59. W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  60. C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the visual interpretation of sentences. ICCV, 2013. 6. Supplementary Material 6.1. Magnitude modulation find