Elamur Research⌕
☰
home / papers / 2016 / Black Holes and White Rabbits: Metaphor Identification with Visual Features

Black Holes and White Rabbits: Metaphor Identification with Visual Features

Summary

The paper addresses automatic metaphor identification: deciding whether short phrases (verb–subject, verb–direct object, adjective–noun) are used literally or metaphorically. Earlier systems relied on text alone. Some also used hand-built resources, such as the MRC concreteness database or WordNet, to capture perceptual properties. The authors instead learn these properties from data in two modalities. Linguistic embeddings for words and phrases come from a skip-gram model trained on parsed Wikipedia. Visual embeddings come from a CNN pretrained on ImageNet and applied to Google Images results for each word or phrase. Simple arithmetic and similarity operations over these vectors produce a metaphoricity score, and a threshold learned from very few labelled examples turns that score into a decision. The multimodal variant outperforms both the text-only and the vision-only models. It comes close to the strongest resource-heavy system (Tsvetkov et al.) and beats many others. This matters because it shows that perceptual grounding helps metaphor detection without hand-annotated lexicons, which makes the approach portable across domains.

Key points

  • Linguistic side: 100-dimensional skip-gram embeddings with negative sampling were trained on an English Wikipedia dump processed with Stanford CoreNLP.
    • Lemmas seen fewer than 100 times were dropped.
    • Training used 3 epochs, a window of 5 and 10 negative samples.
    • Phrases were extracted via nsubj/dobj (verb–noun) and amod (adjective–noun) dependency arcs.
  • Visual side: representations come from the 4096-dimensional FC7 layer of a Krizhevsky-style CNN trained on ImageNet and run in Caffe. Up to 10 Google Images were used per word or phrase.
  • Training data: thresholds were fitted on only 80 annotated examples. Re-training on growing portions of TSV-TRAIN (roughly 170 examples per step) showed the thresholds were stable, with standard deviations of 0.03 for MixLate, 0.02 for WordCos (linguistic) and 0.05 for PhraseCos1 (visual).
  • Results on TSV-TRAIN: the multimodal MixLate reached an F-score of 0.71 and beat the language-only and vision-only models. Scores were lower than on TSV-TEST, which the authors attribute to less consistent labels in TSV-TRAIN.
  • Comparison with prior work on TSV-TEST: the method scores slightly below Tsvetkov et al. (2014), who report an F-score of 0.85 on adjective–noun pairs using MRC and WordNet features. It compares favourably with Turney et al. (2011), who report F 0.68 on verb–noun pairs.

Interesting ideas

  • I1. Pseudo-annotation from image search: retrieving images for a word or phrase and averaging CNN features gives a data-driven stand-in for concreteness and imageability norms. This is attractive for languages such as Ukrainian, where such psycholinguistic databases are scarce or missing.
  • I2. Phrase vectors in the word space: phrase embeddings are learned in a second pass over the corpus while the context vectors from the first pass stay frozen. Phrase and word vectors therefore live in the same space and can be compared directly, which is a cheap trick for any task contrasting a word with its composition.
  • I3. Compositional mismatch as a signal: a phrase is flagged as metaphorical by comparing the vectors of its parts with each other or with the phrase vector, without training a classifier. This kind of unsupervised mismatch score could be reused for detecting idioms or unusual collocations.
  • I4. Small-data thresholding plus a stability check: because the knowledge sits in pretrained embeddings, only a decision threshold is learned from a handful of examples. Checking how much that threshold drifts as training data grows is a useful, inexpensive way to prove a low-resource method is robust.
  • I5. Evaluating on the cleaner test set: the authors note that a set labelled by several high-agreement annotators gives more trustworthy numbers than a larger, noisier one. This is a good reminder to report annotation quality alongside benchmark scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

74 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1020

@inproceedings{shutova2016black,
  title = {{Black Holes and White Rabbits: Metaphor Identification with Visual Features}},
  author = {Shutova, Ekaterina and Kiela, Douwe and Maillard, Jean},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1020},
  url = {https://doi.org/10.18653/v1/n16-1020}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lawrence W. Barsalou. 2008. Grounded cognition. Annual Review of Psychology, 59(1):617–645. find
  2. Beata Beigman Klebanov, Chee Wee Leong, and Michael Flor. 2015. Supervised word-level metaphor detection: Experiments with concreteness and reweighting of examples. In Proceedings of the Third Workshop on Metaphor in NLP, pages 11–20, Denver, Colorado, June. Association for Computational Linguistics… find
  3. Shane Bergsma and Randy Goebel. 2011. Using visual information to predict lexical preference. In RANLP, pages 399–405. find
  4. Danushka Bollegala and Ekaterina Shutova. 2013. Metaphor interpretation using paraphrases extracted from the web. PLoS ONE, 8(9):e74304. find
  5. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res.(JAIR), 49:1–47. find
  6. Jonathan Dunn. 2013a. Evaluating the premises and results of four metaphor identification systems. In Proceedings of CICLing’13, pages 471–486, Samos, Greece. find
  7. Jonathan Dunn. 2013b. What metaphor identification systems can tell us about metaphor-in-language. In Proceedings of the First Workshop on Metaphor in NLP, pages 1–10, Atlanta, Georgia. find
  8. Jerome Feldman. 2006. From Molecule to Metaphor: A Neural Theory of Language. The MIT Press. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database (ISBN: 0-262-06197-X). MIT Press, first edition. find
  9. Robert Fergus, Li Fei-Fei, Pietro Perona, and Andrew Zisserman. 2005. Learning object categories from google’s image search. In Computer Vision, 2005. find
  10. ICCV 2005. Tenth IEEE International Conference on, volume 2, pages 1816–1823. IEEE. find
  11. Lisa Gandy, Nadji Allan, Mark Atallah, Ophir Frieder, Newton Howard, Sergey Kanareykin, Moshe Koppel, Mark Last, Yair Neuman, and Shlomo Argamon. 2013. Automatic identification of conceptual metaphors with limited knowledge. In Proceedings of AAAI 2013. find
  12. Matt Gedigian, John Bryant, Srini Narayanan, and Branimir Ciric. 2006. Catching metaphors. In In Proceedings of the 3rd Workshop on Scalable Natural Language Understanding, pages 41–48, New York. find
  13. Ilana Heintz, Ryan Gabbard, Mahesh Srivastava, Dave Barner, Donald Black, Majorie Friedman, and Ralph Weischedel. 2013. Automatic extraction of linguistic metaphors with lda topic modeling. In Proceedings of the First Workshop on Metaphor in NLP, pages 58–66, Atlanta, Georgia. find
  14. Felix Hill, Anna Korhonen, and Christian Bentz. 2014. A quantitative empirical analysis of the abstract/concrete distinction. Cognitive Science, 38(1):162–177. find
  15. Dirk Hovy, Shashank Shrivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52–57, Atlanta, Georgia. find
  16. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. Douwe Kiela and Léon Bottou. 2014. Learning Image Embeddings using Con… find our card
  17. Douwe Kiela and Stephen Clark. 2015. Multi- and cross-modal semantics beyond vision: Grounding in auditory perception. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2461–2470, Lisbon, Portugal, September. Association for Computational Linguistics. D… find
  18. Saisuresh Krishnakumaran and Xiaojin Zhu. 2007. Hunting elusive metaphors using lexical resources. In Proceedings of the Workshop on Computational Approaches to Figurative Language, pages 13–20, Rochester, NY. find
  19. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. find
  20. George Lakoff and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press, Chicago. find
  21. Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, Michigan, June. find our card
  22. Hongsong Li, Kenny Q. Zhu, and Haixun Wang. 2013. Data-driven metaphor recognition and explanation. Transactions of the Association for Computational Linguistics, 1:379–390. find
  23. Max M Louwerse. 2011. Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, 3(2):273–302. find
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  25. Zachary Mason. 2004. Cormet: a computational, corpus-based conventional metaphor extraction system. Computational Linguistics, 30(1):23–44. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of ICLR, Scottsdale, AZ. find
  27. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of NAACL-HLT, pages 746–751. find our card
  28. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. Language Resources and Evaluation, forthcoming. find our card
  29. Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27–35, Atlanta, Georgia. find
  30. Michael Mohler, Bryan Rink, David Bracewell, and Marc Tomlinson. 2014. A novel distributional approach to multilingual conceptual metaphor recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland. find
  31. Yair Neuman, Dan Assaf, Yohai Cohen, Mark Last, Shlomo Argamon, Newton Howard, and Ophir Frieder. 2013. Metaphor identification in large texts corpora. PLoS ONE, 8(4):e62343. Stephen Roller and Sabine Schulte im Walde. 2013. A multimodal lda model integrating textual, cognitive and visual modalities… find
  32. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–25… find our card
  33. Ekaterina Shutova and Lin Sun. 2013. Unsupervised metaphor identification using hierarchical graph factorization clustering. In Proceedings of NAACL 2013, Atlanta, GA, USA. find
  34. Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of Coling 2010, pages 1002–1010, Beijing, China. find our card
  35. Ekaterina Shutova, Tim Van de Cruys, and Anna Korhonen. 2012. Unsupervised metaphor paraphrasing using a vector space model. In Proceedings of COLING 2012, Mumbai, India. find
  36. Ekaterina Shutova. 2010. Automatic metaphor interpretation as a paraphrasing task. In Proceedings of NAACL 2010, pages 1029–1037, Los Angeles, USA. find
  37. Ekaterina Shutova. 2015. Design and Evaluation of Metaphor Processing Systems. Computational Linguistics, 41(4). find
  38. Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423–1433. Association for Computational Linguistics. find
  39. Tomek Strzalkowski, George Aaron Broadwell, Sarah Taylor, Laurie Feldman, Samira Shaikh, Ting Liu, Boris Yamrom, Kit Cho, Umit Boz, Ignacio Cases, and Kyle Elliot. 2013. Robust extraction of metaphor from novel data. In Proceedings of the First Workshop on Metaphor in NLP, pages 67–76, Atlanta, Geor… find
  40. Yulia Tsvetkov, Elena Mukomel, and Anatole Gershman. 2013. Cross-lingual metaphor detection using common semantic features. In Proceedings of the First Workshop on Metaphor in NLP, pages 45–51, Atlanta, Georgia. find
  41. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 248–258, Baltimore, Maryland, June. Ass… find our card
  42. Peter D. Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 680–690, Stroudsburg, PA, USA. Association for C… find our card
  43. Tony Veale and Yanfen Hao. 2008. A fluid knowledge representation for understanding and generating creative metaphors. In Proceedings of COLING 2008, pages 945–952, Manchester, UK. find
  44. Yorick Wilks, Adam Dalton, James Allen, and Lucian Galescu. 2013. Automatic metaphor detection using large-scale lexical resources and conventional metaphor extraction. In Proceedings of the First Workshop on Metaphor in NLP, pages 36–44, Atlanta, Georgia. find
  45. M.D. Wilson. 1988. The MRC Psycholinguistic Database: Machine Readable Dictionary, Version 2. Behavioural Research Methods, Instruments and Computers, 20:6–11. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shutova-2016-black-holes-white-rabbits-metaphor/