Elamur Research⌕
☰
home / papers / 2015 / Order-Embeddings of Images and Language

Order-Embeddings of Images and Language

Paper · 2015 · Ivan Vendrov, Ryan Kiros, Sanja Fidler et al.

Summary

The authors argue that hypernymy between words, entailment between sentences, and the link between images and their captions are all instances of one partial order, which they call the visual-semantic hierarchy. In this order, general concepts sit above specific ones, and the relation is transitive and antisymmetric. Common embedding approaches compare items with symmetric measures such as cosine distance, so they cannot express which item is more general.

The paper proposes order-embeddings: nonnegative vectors compared under a reversed product order. The origin is the most general concept, and an item counts as more specific when every coordinate is at least as large. Violations of this order are penalized softly inside a max-margin loss, and the penalty can replace the comparison function in existing models.

Trained only on the WordNet hierarchy, the full model was the only variant that beat a transitive-closure baseline on withheld hypernym pairs. The authors also report better caption-image retrieval and near state-of-the-art natural language inference. The broader point is that hierarchy is built into the geometry rather than left for models to learn from data.

Key points

  • The order-violation penalty is E(x, y) = ‖max(0, y − x)‖². It equals zero exactly when the pair satisfies the reversed product order on R^N₊.
  • The hypernym data is the transitive closure of WordNet: 838,073 edges among 82,192 concepts. As in Bordes et al. (2011), 4,000 edges are held out for test and another 4,000 for development.
  • Training setup:
    • 50-dimensional nonnegative vectors, one per concept, stored as a lookup table.
    • Margin α = 1, with 500 true and 500 corrupted pairs per batch.
    • Adam optimizer at learning rate 0.01, 30–50 epochs, with early stopping.
  • Comparisons on WordNet:
    • Baselines: transitive closure (no learning) and word2gauss (Vilnis & McCallum, 2015), which uses Gaussian embeddings with KL divergence.
    • Ablations: the same model with a symmetric cosine comparison, and with a bilinear comparison as in Socher et al. (2013).
    • Only the full asymmetric penalty surpassed the transitive-closure baseline.
  • The supplementary material reports a new regularity in embeddings learned on COCO caption-image retrieval: elementwise max roughly corresponds to composition and elementwise min to abstraction. For example, max("man", "cat") retrieves images showing both.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada110 works · 179,061 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Ivan Vendrov, Ryan Kiros, Sanja Fidler and Raquel Urtasun. 2015. Order-Embeddings of Images and Language. arXiv:1511.06361.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06361 · arXiv 1511.06361

@misc{vendrov2015order,
  title = {{Order-Embeddings of Images and Language}},
  author = {Ivan Vendrov, Ryan Kiros, Sanja Fidler and Raquel Urtasun},
  year = {2015},
  doi = {10.48550/arxiv.1511.06361},
  eprint = {1511.06361},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06361}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baroni, Marco, Bernardi, Raffaella, Do, Ngoc-Quynh, and Shan, Chung-chieh. Entailment above the word level in distributional semantics. In EACL, 2012.
  2. Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua. Learning structured embeddings of knowledge bases. In AAAI, 2011. find
  3. Bowman, Samuel R., Angeli, Gabor, Potts, Christopher, and Manning, Christopher D. A large annotated corpus for learning natural language inference. In EMNLP, 2015.
  4. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. — Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
  5. Chopra, Sumit, Hadsell, Raia, and LeCun, Yann. Learning a similarity metric discriminatively, with application to face verification. In CVPR, 2005. find
  6. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. find
  7. Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013. find
  8. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV, 2015. find
  9. Hodosh, Micah, Young, Peter, and Hockenmaier, Julia. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 2013. find
  10. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015. find
  11. Karpathy, Andrej and Li, Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
  12. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. In ICLR, 2015.
  13. Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.
  14. Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S, Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja. Skip-thought vectors. NIPS, 2015. find
  15. Klein, Benjamin, Lev, Guy, Sadeh, Gil, and Wolf, Lior. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, 2015. find
  16. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  17. Lin, Dahua, Fidler, Sanja, Kong, Chen, and Urtasun, Raquel. Visual semantic search: Retrieving videos via complex textual queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014a. find
  18. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. Microsoft coco: Common objects in context. In ECCV, 2014b.
  19. Ma, Lin, Lu, Zhengdong, Shang, Lifeng, and Li, Hang. Multimodal convolutional neural networks for matching image and sentence. ICCV, 2015. find
  20. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan. Deep captioning with multimodal recurrent neural networks (m-rnn). In ICLR, 2015.
  21. Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey. Linguistic regularities in continuous space word representations. In HLT-NAACL, pp. 746–751, 2013.
  22. Miller, George A. Wordnet: a lexical database for english. Communications of the ACM, 1995.
  23. Norouzi, Mohammad, Mikolov, Tomas, Bengio, Samy, Singer, Yoram, Shlens, Jonathon, Frome, Andrea, Corrado, Greg S, and Dean, Jeffrey. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014. find
  24. Plummer, Bryan, Wang, Liwei, Cervantes, Chris, Caicedo, Juan, Hockenmaier, Julia, and Lazebnik, Svetlana. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. arXiv preprint arXiv:1505.04870, 2015. find
  25. Rocktäschel, Tim, Grefenstette, Edward, Hermann, Karl Moritz, Kočiskỳ, Tomáš, and Blunsom, Phil. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664, 2015.
  26. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  27. Socher, Richard, Chen, Danqi, Manning, Christopher D, and Ng, Andrew. Reasoning with neural tensor networks for knowledge base completion. In NIPS, 2013.
  28. Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y. Grounded compositional semantics for finding and describing images with sentences. TACL, 2014.
  29. Van der Maaten, Laurens and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85, 2008. find
  30. Vilnis, Luke and McCallum, Andrew. Word representations via gaussian embedding. In ICLR, 2015. find
  31. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In CVPR, 2015.
  32. Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014. find
  33. S UPPLEMENTARY M ATERIAL Mikolov et al. (2013) showed that word representations learned using word2vec exhibit semantic regularities, such as king − man + woman ∼ queen. find
  34. Kiros et al. (2014) showed that similar regularities hold for joint image-language models. We find that order-embeddings exhibit a novel form of regularity, shown in Figure 4. The elementwise max and min operations in the embedding space roughly correspond to composition and abstraction, respectivel… find
  35. Query max(“man”, “cat”) max(“black dog”, “park”) , ( ) max , ( ) min , ( ) “dog” min , ( ) “man” max Figure 4: Multimodal regularities found with embeddings learned for the caption-image retrieval task. Note that some images have been slightly cropped for easier viewing, but no relevant objects have… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/