Elamur Research
☰
home / papers / 2014 / Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1411.4389

Summary

Deep CNNs had come to dominate image interpretation, but most video and description systems still relied on a fixed visual representation or simple averaging over time. The authors propose Long-term Recurrent Convolutional Networks (LRCNs). These combine a convolutional visual encoder with stacked LSTM layers, and the whole model is trained end-to-end. Both parts share weights across time steps, so the model handles inputs and outputs of variable length. This makes it "doubly deep": it learns structure across both space and time. The framework is applied to three settings:

  • Video activity recognition: CNN features from each frame feed deep LSTMs.
  • Image captioning: a convnet encodes the image and an LSTM decodes it into a sentence, as a multimodal counterpart to LSTM encoder-decoder translation.
  • Video description: an LSTM decoder generates text from the outputs of conventional semantic role tuple predictors.

The joint models outperform approaches whose recognition and generation components are defined or optimized separately. The work matters because it was an early, general demonstration that recurrent networks attached to convnets can turn pixels directly into language.

Key points

  • LRCN pipeline: a CNN extracts features from each frame or image, a stack of LSTMs models the sequence, and the model emits a variable-length prediction.
  • The paper uses the LSTM variant from [13]. It has input, forget, output and input-modulation gates plus a memory cell, which counters the vanishing-gradient problem of plain RNNs.
  • On standard activity benchmarks, adding LSTMs gives notable gains, even though those datasets show limited temporal complexity.
  • Captioning is end-to-end trainable on large image–text datasets. Even with modest training, it produces generation results competitive with existing methods.
  • For video description, the LSTM decoder beats earlier pipelines based on statistical machine translation. The RNN and LSTM implementations were released in Caffe (jeffdonahue.com/lrcn).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

130 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1411.4389 · arXiv 1411.4389

@misc{donahue2014long,
  title = {{Long-term Recurrent Convolutional Networks for Visual Recognition and Description}},
  author = {Donahue, Jeff and Hendricks, Lisa Anne and Rohrbach, Marcus and Venugopalan, Subhashini and Guadarrama, Sergio and Saenko, Kate and Darrell, Trevor},
  year = {2014},
  doi = {10.48550/arxiv.1411.4389},
  eprint = {1411.4389},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1411.4389}
}

References

73 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks for human action recognition,” in IEEE Trans. Pattern Anal. Mach. Intell., 2013. find
  2. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt, “Sequential deep learning for human action recognition,” in Human Behavior Understanding, 2011. find
  3. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in CVPR, 2014. find
  4. K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in NIPS, 2014. find
  5. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” DTIC Document, Tech. Rep., 1985. find
  6. R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” in Neural Computation, 1989. find
  7. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” in Neural Computation. MIT Press, 1997. find
  8. A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in ICML, 2014. find
  9. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in NIPS, 2014. find
  10. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” in SSST Workshop, 2014. find our card
  11. M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele, “Translating video content to natural language descriptions,” in ICCV, 2013. find
  12. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in ACM MM, 2014. find our card
  13. W. Zaremba and I. Sutskever, “Learning to execute,” in arXiv preprint arXiv:1410.4615, 2014. find
  14. A. Graves, “Generating sequences with recurrent neural networks,” in arXiv preprint arXiv:1308.0850, 2013. find
  15. O. Vinyals, S. V. Ravuri, and D. Povey, “Revisiting recurrent neural networks for robust ASR,” in ICASSP, 2012. find
  16. I. Sutskever, J. Martens, and G. E. Hinton, “Generating text with recurrent neural networks,” in ICML, 2011. find
  17. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in NIPS, 2012. find
  18. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in ICLR, 2015. find our card
  19. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in CVPR, 2015. find
  20. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in EMNLP, 2014. find
  21. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in ECCV, 2004. find
  22. M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in ECCV, 2014. find our card
  23. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” in IJCV, vol. 115, no. 3, 2015. find our card
  24. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in CVPR, 2009. find
  25. K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” CRCV-TR-12-01, Tech. Rep., 2012. find
  26. P. Y. Micah Hodosh and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” in JAIR, vol. 47, 2013. find
  27. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille, “Deep captioning with multimodal recurrent neural networks (m-RNN),” in ICLR, 2015. find
  28. A. Karpathy, A. Joulin, and L. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in NIPS, 2014. find
  29. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, “Grounded compositional semantics for finding and describing images with sentences,” in TACL, vol. 2, 2014. find our card
  30. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov et al., “Devise: A deep visual-semantic embedding model,” in NIPS, 2013. find
  31. R. Kiros, R. Salakhuditnov, and R. S. Zemel, “Unifying visual-semantic embeddings with multimodal neural language models,” in TACL, 2015. find our card
  32. M. H. Peter Young, Alice Lai and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” in TACL, vol. 2, 2014. find
  33. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” arXiv preprint arXiv:1405.0312, Tech. Rep., 2014. find our card
  34. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in ACL, 2002. find our card
  35. R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in CVPR, 2015. find our card
  36. S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005. find our card
  37. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 2004. find
  38. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR, 2015. find our card
  39. J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell, “Language models for image captioning: The quirks and what works,” in ACL, 2015. find
  40. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, “Learning like a child: Fast novel visual concept learning from sentence descriptions of images,” in ICCV, 2015. find
  41. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt et al., “From captions to visual concepts and back,” in CVPR, 2015. find
  42. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, “Exploring nearest neighbor approaches for image captioning,” arXiv preprint arXiv:1505.04467, Tech. Rep., 2015. — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) find
  43. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in CVPR, 2015. find
  44. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in ICML, 2015. find our card
  45. A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in CVPR, 2015. find our card
  46. P. Kuznetsova, V. Ordonez, T. L. Berg, U. C. Hill, and Y. Choi, “TreeTalk: Composition and compression of trees for image descriptions,” in TACL, vol. 2, no. 10, 2014. find
  47. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in ACL, 2007. find
  48. A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, and B. Schiele, “Coherent multi-sentence video description with variable level of detail,” in German Conference on Pattern Recognition (GCPR). Springer, 2014. find
  49. H. Wang, A. Kläser, C. Schmid, and C. Liu, “Dense trajectories and motion boundary descriptors for action recognition,” in IJCV, 2013. find
  50. H. Wang and C. Schmid, “Action recognition with improved trajectories,” in ICCV, 2013. find
  51. H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “HMDB: a large video database for human motion recognition,” in ICCV, 2011. find
  52. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt, “Action classification in soccer videos with long short-term memory recurrent neural networks,” in International Conference on Artificial Neural Networks (ICANN), 2010. find
  53. A. Farhadi, M. Hejrati, M. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in ECCV, 2010. find
  54. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Baby talk: Understanding and generating simple image descriptions,” in CVPR, 2011. find
  55. Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos, “Corpus-guided sentence generation of natural images,” in EMNLP, 2011. find
  56. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III, “Midge: Generating image descriptions from computer vision detections,” in Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, 2… find
  57. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, “Collective generation of natural image descriptions,” in ACL, 2012. find
  58. R. Kiros, R. Salakhutdinov, and R. Zemel, “Multimodal neural language models,” in ICML, 2014. find
  59. S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko, “YouTube2Text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shoot recognition,” in ICCV, 2013. find
  60. M. U. G. Khan, L. Zhang, and Y. Gotoh, “Human focused video description,” in ICCV Workshops, 2011. find
  61. A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy, D. Salvi, L. Schmidt, J. Shangguan, J. M. Siskind, J. Waggoner, S. Wang, J. Wei, Y. Yin, and Z. Zhang, “Video in sentences out,” in The Conference on Uncertainty in Artificial Intelligenc… find
  62. P. Das, C. Xu, R. Doell, and J. Corso, “Thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching,” in CVPR, 2013. find
  63. C. C. Tan, Y.-G. Jiang, and C.-W. Ngo, “Towards textually describing complex video contents with audio-visual concept classifiers,” in ACM MM, 2011. find
  64. J. Thomason, S. Venugopalan, S. Guadarrama, K. Saenko, and R. J. Mooney, “Integrating language and vision to generate natural language descriptions of videos in the wild,” in International Conference on Computational Linguistics (COLING), 2014. find
  65. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao, “Sequence discriminative distributed training of long short-term memory recurrent neural networks,” in Interspeech, 2014. find
  66. J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in CVPR, 2015. find
  67. S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei, “Every moment counts: Dense detailed labeling of actions in complex videos,” arXiv preprint arXiv:1507.05738, Tech. Rep., 2015. — [arXiv:1507.05738](https://arxiv.org/abs/1507.05738) find
  68. L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, “Deep compositional captioning: Describing novel object categories without paired training data,” in CVPR, 2016. find
  69. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko, “Translating videos to natural language using deep recurrent neural networks,” in NAACL, 2015. find
  70. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, “Sequence to sequence–video to text,” in ICCV, 2015. find
  71. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville, “Describing videos by exploiting temporal structure,” in CVPR, vol. 1050, 2015. find
  72. A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele, “Grounding of textual phrases in images by reconstruction,” arXiv preprint arXiv:1511.03745, Tech. Rep., 2015. — [arXiv:1511.03745](https://arxiv.org/abs/1511.03745) find
  73. R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell, “Natural language object retrieval,” in CVPR, 2016. Jeff Donahue is a PhD student at the University of California, Berkeley, advised by Prof. Trevor Darrell. His research focuses on the use of deep learning for computer vision applicatio… find