Elamur Research⌕
☰
home / papers / 2014 / Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Paper · 2014 · Junhua Mao, Wei Xu, Yi Yang et al.

Summary

The paper addresses automatic image captioning. Many earlier systems picked captions from a fixed sentence database and so could not describe unseen combinations of objects and scenes. The authors propose m-RNN, a generative model that estimates the probability of each next word given the preceding words and the image. Captions are produced by sampling from this distribution.

The architecture has three parts:

  • a language branch with two word embedding layers and a recurrent layer;
  • a deep CNN that encodes the image;
  • a single multimodal layer that fuses the two.

The image features enter the multimodal layer at every time step rather than being stored in the recurrent state. The whole network is trained jointly with a log-likelihood objective, and word embeddings are learned from random initialization.

Experiments on IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO show state-of-the-art results for caption generation. The model also performs well on image-to-sentence and sentence-to-image retrieval, beating methods built specifically around ranking losses. This matters because a single probabilistic model handles both generation and retrieval.

Key points

  • Each time frame is built from five layers (a pair of word-embedding layers followed by recurrent, multimodal and softmax layers) together with a deep CNN. Weights are shared across time steps.
  • Ablations over seven variants show two findings:
    • The two-layer embedding (layer II feeding the multimodal layer) beats a single 256-dimensional embedding while using fewer parameters (128·M + 128·256 vs 256·M).
    • Feeding visual features directly to the multimodal layer outperforms injecting them into the recurrent part, and the recurrent-injection variants were also harder to train.
  • On IAPR TC-12 retrieval, m-RNN clearly beats bow-decaf, MLBL-F-decaf and MLBL-B-decaf, using the 100-image shortlist protocol of Kiros et al.
  • BLEU (B-1 to B-4) is computed at the corpus level, following Fang et al.
  • Code, refined image features and generated hypotheses are released at github.com/mjhucla/mRNN-CR. An earlier version appeared at the NIPS 2014 Deep Learning Workshop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China84 works · 17,681 citations🎓University of California, Los AngelesLos Angeles · United States99 works · 56,330 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang and Alan Yuille. 2014. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). arXiv:1412.6632 · ICLR 2015.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1412.6632 · arXiv 1412.6632

@misc{mao2014deep,
  title = {{Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)}},
  author = {Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang and Alan Yuille},
  year = {2014},
  doi = {10.48550/arxiv.1412.6632},
  eprint = {1412.6632},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.6632}
}

References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barnard, Kobus, Duygulu, Pinar, Forsyth, David, De Freitas, Nando, Blei, David M, and Jordan, Michael I. Matching words and pictures. JMLR, 3:1107–1135, 2003. find
  2. Chen, X., Fang, H., Lin, TY, Vedantam, R., Gupta, S., Dollr, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  3. Chen, Xinlei and Zitnick, C Lawrence. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014. find
  4. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  5. Devlin, Jacob, Cheng, Hao, Fang, Hao, Gupta, Saurabh, Deng, Li, He, Xiaodong, Zweig, Geoffrey, and Mitchell, Margaret. Language models for image captioning: The quirks and what works. arXiv preprint arXiv:1505.01809, 2015a. find
  6. Devlin, Jacob, Gupta, Saurabh, Girshick, Ross, Mitchell, Margaret, and Zitnick, C Lawrence. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467, 2015b. find
  7. Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio, Rohrbach, Marcus, Venugopalan, Subhashini, Saenko, Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411.4389, 2014.
  8. Elman, Jeffrey L. Finding structure in time. Cognitive science, 14(2):179–211, 1990. find
  9. Fang, Hao, Gupta, Saurabh, Iandola, Forrest, Srivastava, Rupesh, Deng, Li, Dollár, Piotr, Gao, Jianfeng, He, Xiaodong, Mitchell, Margaret, Platt, John, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014.
  10. Farhadi, Ali, Hejrati, Mohsen, Sadeghi, Mohammad Amin, Young, Peter, Rashtchian, Cyrus, Hockenmaier, Julia, and Forsyth, David. Every picture tells a story: Generating sentences from images. In ECCV, pp. 15–29. 2010. find
  11. Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al. Devise: A deep visual-semantic embedding model. In NIPS, pp. 2121–2129, 2013. find
  12. Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  13. Grubinger, Michael, Clough, Paul, Müller, Henning, and Deselaers, Thomas. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International Workshop OntoImage, pp. 13–23, 2006. find
  14. Guillaumin, Matthieu, Verbeek, Jakob, and Schmid, Cordelia. Multiple instance metric learning from automatically labeled bags of faces. In ECCV, pp. 634–647, 2010. find
  15. Gupta, Ankush and Mannem, Prashanth. From image annotation to image description. In ICONIP, 2012. find
  16. Gupta, Ankush, Verma, Yashaswi, and Jawahar, CV. Choosing linguistics over vision to describe images. In AAAI, 2012. find
  17. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  18. Hodosh, Micah, Young, Peter, and Hockenmaier, Julia. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47:853–899, 2013. find
  19. Jia, Yangqing, Salzmann, Mathieu, and Darrell, Trevor. Learning cross-modality similarity for multinomial data. In ICCV, pp. 2407–2414, 2011. find
  20. Kalchbrenner, Nal and Blunsom, Phil. Recurrent continuous translation models. In EMNLP, pp. 1700–1709, 2013.
  21. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2014.
  22. Karpathy, Andrej, Joulin, Armand, and Fei-Fei, Li. Deep fragment embeddings for bidirectional image sentence mapping. In arXiv:1406.5679, 2014. find
  23. Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014a.
  24. Kiros, Ryan, Zemel, R, and Salakhutdinov, Ruslan. Multimodal neural language models. In ICML, 2014b. find
  25. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, pp. 1097–1105, 2012.
  26. Kulkarni, Girish, Premraj, Visruth, Dhar, Sagnik, Li, Siming, Choi, Yejin, Berg, Alexander C, and Berg, Tamara L. Baby talk: Understanding and generating image descriptions. In CVPR, 2011. find
  27. Kuznetsova, Polina, Ordonez, Vicente, Berg, Tamara L, and Choi, Yejin. Treetalk: Composition and compression of trees for image descriptions. Transactions of the Association for Computational Linguistics, 2(10):351–362, 2014.
  28. LeCun, Yann A, Bottou, Léon, Orr, Genevieve B, and Müller, Klaus-Robert. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9–48. Springer, 2012. find
  29. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2014.
  30. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan L. Explain images with multimodal recurrent neural networks. NIPS DeepLearning Workshop, 2014. find
  31. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, Huang, Zhiheng, and Yuille, Alan. Learning like a child: Fast novel visual concept learning from sentence descriptions of images. arXiv preprint arXiv:1504.06692, 2015. — [arXiv:1504.06692](https://arxiv.org/abs/1504.06692) find
  32. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, pp. 1045–1048, 2010. find
  33. Mikolov, Tomas, Kombrink, Stefan, Burget, Lukas, Cernocky, JH, and Khudanpur, Sanjeev. Extensions of recurrent neural network language model. In ICASSP, pp. 5528–5531, 2011. find
  34. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111–3119, 2013.
  35. Mitchell, Margaret, Han, Xufeng, Dodge, Jesse, Mensch, Alyssa, Goyal, Amit, Berg, Alex, Yamaguchi, Kota, Berg, Tamara, Stratos, Karl, and Daumé III, Hal. Midge: Generating image descriptions from computer vision detections. In EACL, 2012. find
  36. Mnih, Andriy and Hinton, Geoffrey. Three new graphical models for statistical language modelling. In ICML, pp. 641–648. ACM, 2007. find
  37. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807–814, 2010. find
  38. Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu, Wei-Jing. Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311–318, 2002.
  39. Rashtchian, Cyrus, Young, Peter, Hodosh, Micah, and Hockenmaier, Julia. Collecting image annotations using amazon’s mechanical turk. In NAACL-HLT workshop 2010, pp. 139–147, 2010. — Collecting Image Annotations Using Amazon's Mechanical Turk
  40. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning representations by back-propagating errors. Cognitive modeling, 1988. find
  41. Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. ImageNet Large Scale Visual Recognition Challenge, 2014.
  42. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  43. Socher, Richard, Le, Q, Manning, C, and Ng, A. Grounded compositional semantics for finding and describing images with sentences. In TACL, 2014.
  44. Srivastava, Nitish and Salakhutdinov, Ruslan. Multimodal learning with deep boltzmann machines. In NIPS, pp. 2222–2230, 2012. find
  45. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In NIPS, pp. 3104–3112, 2014.
  46. Vedantam, Ramakrishna, Zitnick, C Lawrence, and Parikh, Devi. Cider: Consensus-based image description evaluation. arXiv preprint arXiv:1411.5726, 2014.
  47. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014.
  48. Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In ACL, pp. 479–488, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/