Elamur Research⌕
☰
home / papers / 2015 / Combining Language and Vision with a Multimodal Skip-gram Model

Combining Language and Vision with a Multimodal Skip-gram Model

Summary

Purely text-based distributional models lack grounding in perception. Earlier multimodal models had three problems. They built text and image vectors separately and fused them afterwards. They needed visual data for every word. As a result, they could not reach words or images outside their training set. The authors extend SKIP-GRAM so that, while it predicts corpus contexts, some target words are also paired with fixed visual vectors taken from natural images. The objective adds a visual term, which is zero for words that have no images.

There are two variants:

  • MMSKIP-GRAM-A pushes a word embedding toward its own image vector using a max-margin cosine loss.
  • MMSKIP-GRAM-B learns a cross-modal mapping matrix jointly with the embeddings.

Because the joint training spreads visual information to words that never had images, the vectors do well on general semantic benchmarks. They also improve zero-shot image labeling and retrieval compared with text-only vectors. In addition, they give plausible visual associations for abstract words. This matters because a single incremental model handles both semantic and vision tasks.

Key points

  • The visual loss is a max-margin hinge on cosine similarity. Negative image vectors are sampled uniformly from the image dictionary, with their number set by hyperparameter k. Training uses hierarchical softmax.

  • In MMSKIP-GRAM-B, an L2-regularized matrix M u→v maps linguistic vectors into visual space. This removes the requirement that both vectors have equal dimensionality.

  • In a CrowdFlower study on the Kiela et al. (2014) word set, subjects preferred the model's nearest image over a random one in 48% of cases overall. The rate was 73% for concrete words and 23% for abstract words. Only two abstract words had direct visual input during training.

  • On the 200 Kiela et al. (2014) words, the entropy of the vectors correlates with concreteness:

    VectorsSpearman ρ
    MMSKIP-GRAM-A−0.75
    MMSKIP-GRAM-B mapped to visual space (B*)−0.71
    Kiela et al. image-dispersion measure−0.65
    SKIP-GRAM0.05
    Unmapped MMSKIP-GRAM-B0.04

Interesting ideas

  • I1. Optional auxiliary loss. Add a second-modality loss only for the tokens that have such data, and set it to zero elsewhere. The shared training then carries the signal to uncovered vocabulary. This fits any setting with partial annotations, such as a small lexicon or sparse metadata.
  • I2. Learned mediating projection. Instead of forcing the embedding to equal the external vector, learn a projection between them. This keeps the word space free in size and lets the same design take in other signals, for example feature norms or brain data.
  • I3. Testing propagation honestly. Report results separately for words that were never exposed to the extra modality during training. This shows whether knowledge truly propagated rather than being memorized.
  • I4. Entropy as an unsupervised proxy for abstractness. Shift vector values to be positive, unit-normalize, and compute entropy. Being cheap, this could help rank words by concreteness in languages that lack rating norms.
  • I5. Human evaluation for abstract words. Show annotators the model's nearest image next to a random foil. This gives a simple way to judge outputs that have no gold labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1016

@inproceedings{lazaridou2015combining,
  title = {{Combining Language and Vision with a Multimodal Skip-gram Model}},
  author = {Lazaridou, Angeliki and Pham, Nghia The and Baroni, Marco},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1016},
  url = {https://doi.org/10.3115/v1/n15-1016}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Eduard Barbu, Brian Murphy, and Massimo Poesio. 2010. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222–254. find
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, pages 238–247, Baltimore, MD. find our card
  3. Lawrence Barsalou and Katja Wiemer-Hastings. 2005. Situating abstract concepts. In D. Pecher and R. Zwaan, editors, Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thought, pages 129–163. Cambridge University Press, Cambridge, UK. find
  4. Lawrence Barsalou. 2008. Grounded cognition. Annual Review of Psychology, 59:617–645. find
  5. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. 2012. Distributional semantics in Technicolor. In Proceedings of ACL, pages 136–145, Jeju Island, Korea. find our card
  6. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. find
  7. Stephen Clark. 2015. Vector space models of lexical meaning. In Shalom Lappin and Chris Fox, editors, Handbook of Contemporary Semantics, 2nd ed. Blackwell, Malden, MA. In press; http://www.cl.cam.ac.uk/˜sc609/pubs/sem_handbook.pdf. ↗
  8. Jia Deng, Wei Dong, Richard Socher, Lia-Ji Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of CVPR, pages 248–255, Miami Beach, FL. find
  9. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  10. Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In Proceedings of CVPR, pages 1778–1785, Miami Beach, FL. find
  11. Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In Proceedings of HLT-NAACL, pages 91–99, Los Angeles, CA. find
  12. Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A deep visual-semantic embedding model. In Proceedings of NIPS, pages 2121– 2129, Lake Tahoe, NV. find
  13. Alona Fyshe, Partha P Talukdar, Brian Murphy, and Tom M Mitchell. 2014. Interpretable semantic vectors from a joint model of brain-and text-based meaning. In In Proceedings of ACL, pages 489–499. find
  14. Arthur Glenberg and David Robertson. 2000. Symbol grounding and meaning: A comparison of high-dimensional and embodied theories of meaning. Journal of Memory and Language, 3(43):379–401. find
  15. Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346. find
  16. Felix Hill and Anna Korhonen. 2014. Learning abstract concept embeddings from multi-modal data: Since you probably can’t see what I mean. In Proceedings of EMNLP, pages 255–265, Doha, Qatar. find
  17. Felix Hill, Roi Reichart, and Anna Korhonen. 2014. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. http://arxiv.org/abs/arXiv:1408.3456. ↗ our card
  18. Steve Howell, Damian Jankowicz, and Suzanna Becker. 2005. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. Journal of Memory and Language, 53:258–276. find
  19. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. find our card
  20. Andrej Karpathy, Armand Joulin, and Li Fei-Fei. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In Proceedings of NIPS, pages 1097–1105, Montreal, Canada. find
  21. Douwe Kiela and Léon Bottou. 2014. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of EMNLP, pages 36–45, Doha, Qatar. find
  22. Douwe Kiela, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, pages 835–841, Baltimore, MD. find
  23. Ryan Kiros, Ruslan Salakhutdinov, and Richard Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. In Proceedings of the NIPS Deep Learning and Representation Learning Workshop, Montreal, Canada. Published online: http://www.dlworkshop.org/accepted-papers. ↗ our card
  24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NIPS, pages 1097– 1105, Lake Tahoe, Nevada. find
  25. George Lakoff and Mark Johnson. 1999. Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. Basic Books, New York. find
  26. Angeliki Lazaridou, Elia Bruni, and Marco Baroni. 2014. Is this a wampimuk? cross-modal mapping between distributional semantics and the visual world. In Proceedings of ACL, pages 1403–1414, Baltimore, MD. find
  27. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. 2014. Explain images with multimodal recurrent neural networks. In Proceedings of the NIPS Deep Learning and Representation Learning Workshop, Montreal, Canada. Published online: http://www.dlworkshop.org/accepted-papers. ↗
  28. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111–3119, Lake Tahoe, NV. find
  29. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of NAACL, pages 746–751, Atlanta, Georgia. find our card
  30. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of AISTATS, pages 246–252, Barbados. find
  31. Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of CVPR. find
  32. John Searle. 1984. Minds, Brains and Science. Harvard University Press, Cambridge, MA. find
  33. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, pages 721–732, Baltimore, Maryland. find
  34. Josef Sivic and Andrew Zisserman. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings of ICCV, pages 1470–1477, Nice, France. find
  35. Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Proceedings of NIPS, pages 935–943, Lake Tahoe, NV. find
  36. Richard Socher, Quoc Le, Christopher Manning, and Andrew Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218. find our card
  37. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  38. Peter Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of EMNLP, pages 680–690, Edinburgh, UK. find our card
  39. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of CHI, pages 319–326, Vienna, Austria. find
  40. Jason Weston, Samy Bengio, and Nicolas Usunier. 2010. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81(1):21–35. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lazaridou-2015-combining-language-vision-multimodal-skip/