Elamur Research⌕
☰
home / papers / 2022 / Testing the Ability of Language Models to Interpret Figurative Language

Testing the Ability of Language Models to Interpret Figurative Language

Paper · 2022 · Emmy Liu, Chenxuan Cui, Kenneth Zheng et al.

Summary

Figurative language is everywhere in everyday speech, yet NLP research has mostly focused on detecting metaphors rather than on whether models can actually work out what a metaphor implies. Liu et al. address this gap with Fig-QA, a crowdsourced English benchmark modeled on the Winograd schema. Each item pairs two creative metaphors or similes that differ only slightly but mean opposite things, such as an object of comparison like plywood versus oak. Each metaphor comes with a literal implication, and the model must pick the one that follows. Amazon Mechanical Turk workers were asked to produce rare, novel expressions that are unlikely to appear in pretraining corpora.

The authors evaluated GPT-2, GPT-neo, GPT-3, BERT and RoBERTa in zero-shot, few-shot and fine-tuned settings. Models scored clearly above chance, and fine-tuning brought them close to good performance. However, zero- and few-shot results lagged well behind humans. Error analysis showed that models largely ignore the metaphorical context and lean on how plausible each answer is in isolation. The benchmark exposes nonliteral reasoning as a persistent weakness of LMs.

Key points

  • Fig-QA contains 10,256 validated metaphor–interpretation items, filtered by three authors from 13,324 crowdsourced submissions. The large train split has 8,016 items, dev has 1,094 and test has 1,146.
  • Zero-shot scoring compares LM probabilities of metaphor–answer concatenations in two directions: forward (answer given phrase) and backward (phrase given answer). Matching a metaphor to its meaning turned out harder than the reverse direction.
  • The test set was annotated for overlapping knowledge types:
    • commonsense object knowledge: 68.35%
    • social understanding: 27.55%
    • cultural, US-centered: 16.56%
    • visual: 14.73%
  • Even GPT-3 makes errors that do not resemble human mistakes. Human accuracy stays high, even though about half of the human participants were non-native English speakers.
  • Code, data and an ExplainaBoard leaderboard are public (github.com/nightingal3/Fig-QA).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksIdioms & multiword expressions · 155 works

Cite

Cite as Emmy Liu, Chenxuan Cui, Kenneth Zheng and Graham Neubig. 2022. Testing the Ability of Language Models to Interpret Figurative Language. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.330

@inproceedings{liu2022testing,
  title = {{Testing the Ability of Language Models to Interpret Figurative Language}},
  author = {Emmy Liu, Chenxuan Cui, Kenneth Zheng and Graham Neubig},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.330},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.330}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. find
  2. Beata Beigman Klebanov, Chee Wee (Ben) Leong, and Michael Flor. 2018. A corpus of non-native written English annotated for metaphor. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pape… find
  3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  4. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lang…
  5. Gilles Fauconnier and Mark Turner. 2003. Conceptual blending, form and meaning. find
  6. Yuri Bizzoni and Mehdi Ghanimifard. 2018. Bigrams and BiLSTMs two neural networks for sequential metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, pages 91–101, New Orleans, Louisiana. Association for Computational Linguistics. find
  7. Susan Fussell and Mallie Moss. 2008. Figurative language in emotional communication. find
  8. Yuri Bizzoni and Shalom Lappin. 2018. Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, pages 45–55, New Orleans, Louisiana. Association for Computational Linguistics. find
  9. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. find
  10. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  11. Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 607–613, Brussels, Belgium. Association for Computational Linguistics.
  12. D Gentnder and B Bowdle. 2008. Metaphor as structure-mapping. In The Cambridge handbook of metaphor and thought, pages 109–128. Cambridge University Press. find
  13. Kahlil Gibran. 1926. Sand and Foam; a book of aphorisms. A.A Knopf. find
  14. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…
  15. Sam Glucksberg. 2003. The psycholinguistics of metaphor. Trends in cognitive sciences, 7:92–96. find
  16. Robyn Carston and Catherine Wearing. 2011. Metaphor, hyperbole and simile: A pragmatic approach. Language and Cognition, 3(2):283–312. find
  17. Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating robustness of dialog models to popular figurative language constructs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7476–7485, Online and Punta Cana, Domini… find
  18. Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2021a. It’s not rocket science : Interpreting figurative language in narratives. ArXiv, abs/2109.00087.
  19. Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. 2021b. Figurative language in recognizing textual entailment. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3354–3361, Online. Association for Computational Linguistics.
  20. Thomas C. Cooper. 1999. Processing of idioms by l2 learners of english. TESOL Quarterly, 33:233–262. find
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  22. Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to square one: Artifact detection, training and commonsense disentanglement in the winograd schema. find
  23. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  24. G. Lakoff and M. Johnson. 1981. Metaphors we Live By. University of Chicago Press. find
  25. Chee Wee (Ben) Leong, Beata Beigman Klebanov, Chris Hamill, Egon Stemle, Rutuja Ubale, and Xianyang Chen. 2020. A report on the 2020 VUA and TOEFL metaphor detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 18–29, Online. Association for Computation…
  26. Chee Wee (Ben) Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 VUA metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56–66, New Orleans, Louisiana. Association for Computational Linguistics.
  27. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, page 552–561. AAAI Press. find
  28. Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  30. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. CoRR, cs.CL/0205028. find
  31. Rui Mao, Chenghua Lin, and Frank Guerin. 2018. Word embedding and WordNet based metaphor identification and interpretation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1222–1231, Melbourne, Australia. Association for Compu… find
  32. J. S Mio and A. N Katz. 1996. Metaphor: Implications and Applications. Psychology Press. find
  33. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33, Berlin, Germany. Association for Computational Linguistics.
  34. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find
  35. Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192–204, Punta Cana, Dominican Re… find
  36. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language…
  37. Malay Pramanick, Ashim Gupta, and Pabitra Mitra. 2018. An LSTM-CRF based approach to token-level metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, pages 67–75, New Orleans, Louisiana. Association for Computational Linguistics. find
  38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  39. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI. find
  40. Ekaterina Shutova. 2010. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1029–1037, Los Angeles, California. Association for Computational Lingui… find
  41. Ekaterina Shutova. 2011. Computational approaches to figurative language. find
  42. Gerard J Steen, Alleta G Dorst, Berenike Herrmann, Anna A Kaal, Tina Krennmayr, and Trynetje Pasma. 2010. A Method for Linguistic Metaphor Identification: From MIP to MIPVU. John Benjamins. find
  43. Kevin Stowe and Martha Palmer. 2018. Leveraging syntactic constructions for metaphor identification. In Proceedings of the Workshop on Figurative Language Processing, pages 17–26, New Orleans, Louisiana. Association for Computational Linguistics. find
  44. Chang Su, Shuman Huang, and Yijiang Chen. 2017. Automatic detection and interpretation of nominal metaphor based on the theory of meaning. Neurocomputing, 219:300–311. find
  45. John Sweller. 2006. Discussion of ’emerging topics in cognitive load research: Using learner and information characteristics in the design of powerful learning environments’. Applied Cognitive Psychology -APPL COGNITIVE PSYCHOL, 20:353–357. find
  46. Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis. 2021. Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective. In NAACL 2021. find
  47. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 248–258, Baltimore, Maryland. Associati…
  48. P. Wolff and Dedre Gentner. 2000. Evidence for role-neutral initial processing of metaphors. Journal of experimental psychology. Learning, memory, and cognition, 26 2:529–41. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2022-testing-ability-language-models-interpret/