Elamur Research
☰
home / papers / 2014 / Grounded Compositional Semantics for Finding and Describing Images with Sentences

Grounded Compositional Semantics for Finding and Describing Images with Sentences

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q14-1017.pdf

Summary

Earlier Recursive Neural Networks (RNNs) produced sentence vectors that captured syntax well but did not reflect what a sentence shows visually. The authors propose the DT-RNN, which builds sentence vectors over dependency trees instead of constituency trees. Because the root of a dependency tree is usually the main verb, the model puts more weight on the action and its participants. It is also less sensitive to word order and to variation such as active versus passive voice.

Word vectors (d = 50, initialized from Huang et al., 2012) are combined along parses from the de Marneffe et al. (2006) dependency parser. Image features come from a convolutional neural network. Both modalities are projected jointly into a shared embedding space.

On the Rashtchian et al. (2010) dataset, the model was tested on retrieving images from sentences and sentences from images. It beat constituency-tree RNNs, recurrent networks, kernelized CCA and a bag-of-words baseline. It also gave closer vectors to different captions of the same image. The work points toward multimodal retrieval and zero-shot learning that goes beyond single labels.

Key points

  • Dataset: 1000 images from Rashtchian et al. (2010), each with 5 human-written descriptions.
  • Image search (SDT-RNN vs. BoW / kCCA):
    • Mean rank 12.5 vs. 14.6 / 15.9 (lower is better).
    • R@5 46.6 vs. 42.2 / 41.4.
    • R@10 65.6 vs. 60.0 / 58.0.
  • Image description (SDT-RNN vs. BoW / kCCA):
    • Mean rank 16.9 vs. 21.1 / 18.0.
    • R@1 23.0 vs. 19.0 / 21.0.
    • R@10 63.0 vs. 57.0 / 61.0.
    • The only metric where SDT-RNN did not win is R@5, where kCCA scored 47.0 against 45.0.
  • Training choices: a max-margin objective with inner products worked better for every model than squared-error loss or Euclidean distance.
  • Analysis:
    • When ranked by Frobenius norm, the highest-scoring composition matrices were those for relations such as nominal subject, possession modifier, passive auxiliary and spatial prepositions.
    • The main failure case was a caption pair where one sentence has a verb and the other does not.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dependency parsing

Cited by

132 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00177

@article{socher2014grounded,
  title = {{Grounded Compositional Semantics for Finding and Describing Images with Sentences}},
  author = {Socher, Richard and Karpathy, Andrej and Le, Quoc V. and Manning, Christopher D. and Ng, Andrew Y.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00177},
  url = {https://doi.org/10.1162/tacl_a_00177}
}

References

42 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Andrew, R. Arora, K. Livescu, and J. Bilmes. 2013. Deep canonical correlation analysis. In ICML, Atlanta, Georgia. find
  2. K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. Jordan. 2003. Matching words and pictures. find
  3. JMLR. M. Baroni and A. Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  4. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. find
  5. F. Costa, P. Frasconi, V. Lombardo, and G. Soda. 2003. Towards incremental parsing of natural language using recursive neural networks. find
  6. Applied Intelligence. M. de Marneffe, B. MacCartney, and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC. find our card
  7. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A.Y. Ng. 2012. Large scale distributed deep networks. In NIPS. find
  8. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. find
  9. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR. J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12, July. find
  10. P. Duygulu, K. Barnard, N. de Freitas, and D. Forsyth. 2002. Object recognition as machine translation. In ECCV. find
  11. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. 2010. Every picture tells a story: Generating sentences from images. In ECCV. find
  12. Y. Feng and M. Lapata. 2013. Automatic caption generation for news images. IEEE Trans. Pattern Anal. Mach. Intell., 35. find
  13. A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov. 2013. Devise: A deep visual-semantic embedding model. In NIPS. C. Goller and A. Küchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of the Internation… find
  14. E. Grefenstette, G. Dinu, Y.-Z. Zhang, M. Sadrzadeh, and M. Baroni. 2013. Multi-step regression learning for compositional distributional semantics. In IWCS. find
  15. A. Gupta and L. S. Davis. 2008. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In ECCV. find
  16. M. Hodosh, P. Young, and J. Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res. (JAIR), 47:853–899. find
  17. E. H. Huang, R. Socher, C. D. Manning, and A. Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. In ACL. find
  18. K. Jarrett, K. Kavukcuoglu, M.A. Ranzato, and Y. Le-Cun. 2009. What is the best multi-stage architecture for object recognition? In ICCV. find
  19. P. Blunsom. K.M. Hermann. 2013. The role of syntax in vector space models of compositional semantics. In ACL. find
  20. A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. find
  21. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. 2011. Baby talk: Understanding and generating image descriptions. In CVPR. find
  22. N. Kumar, A. C. Berg, P. N. Belhumeur, , and S. K. Na-yar. 2009. Attribute and simile classifiers for face verification. In ICCV. find
  23. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Yejin Choi. 2012. Collective generation of natural image descriptions. In ACL. find
  24. Q. V. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, and A. Y. Ng. 2012. Building high-level features using large scale unsupervised learning. In ICML. find
  25. T. Mikolov, W. Yih, and G. Zweig. 2013. Linguistic regularities in continuous spaceword representations. In HLT-NAACL. find
  26. J. Mitchell and M. Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  27. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A.Y. Ng. 2011. Multimodal deep learning. In ICML. find
  28. V. Ordonez, G. Kulkarni, and T. L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In NIPS. find
  29. J. B. Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46, November. find
  30. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Workshop on Creating Speech and Language Data with Amazon’s MTurk. find
  31. R. Socher and L. Fei-Fei. 2010. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In CVPR. find
  32. R. Socher, C. D. Manning, and A. Y. Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop. find
  33. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. 2011a. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS. find
  34. R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. 2011b. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. find
  35. R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. 2011c. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In EMNLP. find our card
  36. R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. 2012. find
  37. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In EMNLP. R. Socher, J. Bauer, C. D. Manning, and A. Y. Ng. 2013a. find
  38. Parsing With Compositional Vector Grammars. In ACL. R. Socher, M. Ganjoo, C. D. Manning, and A. Y. Ng. 2013b. Zero-Shot Learning Through Cross-Modal Transfer. In NIPS. R. Socher, M. Ganjoo, H. Sridhar, O. Bastani, and A. Y. Ng. C. D. Manning and. 2013c. Zero-shot learning through cross-modal transfe… find
  39. N. Srivastava and R. Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In NIPS. find
  40. A. Torralba, K. P. Murphy, and W. T. Freeman. 2010. Using the forest to see the trees: exploiting context for visual object detection and localization. Communications of the ACM. find
  41. P. D. Turney and P. Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  42. B. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. 2010. I2t:image parsing to text description. IEEE Xplore. find