Elamur Research⌕
☰
home / papers / 2021 / Figurative Language in Recognizing Textual Entailment

Figurative Language in Recognizing Textual Entailment

Paper · 2021 · Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak et al.

Summary

The paper asks whether entailment models trained on standard NLI data can handle non-literal language. The authors treat interpreting a figure of speech as an entailment problem: the figurative sentence is the premise and its intended literal meaning is the hypothesis. They convert five existing resources covering similes, metaphors and irony into more than 12,500 RTE pairs. For similes and metaphors, they reuse human-written literal–figurative paraphrase pairs as entailed examples. They build non-entailed counterparts by manually replacing the key literal word with an antonym. For irony, they use parallel ironic/intended-meaning corpora and also recast irony-labelled tweets into hypotheses through templates. They then test NBoW, InferSent and RoBERTa-large, all trained on MNLI. RoBERTa-large does fairly well wherever lexical antonyms or negation reveal the meaning. It fails when interpretation depends on pragmatic inference, perceptual or world knowledge, unconventional metaphors, or recognizing that a speaker is being ironic. The resulting test suite exposes gaps that headline NLI scores hide.

Key points

  • Test set sizes:
    • Simile: 600 pairs (300 E / 300 NE).
    • Metaphor: 613 pairs (307 / 306).
    • SIGN 2000: 2,000 annotated pairs from Peled & Reichart's corpus (133 E / 1,867 NE).
    • S im -H int (Ghosh et al. 2020): 4,762 pairs, all NE.
    • Irony Intention, from Van Hee et al. 2018 tweets: 4,601 pairs.
  • Ironic-intent pairs are generated with the templates "Name tweeted that tweet" / "Name was ironic". Names are sampled according to US census data.
  • RoBERTa-large (MNLI) accuracy:
    • Simile: 85.47.
    • Metaphor: 88.09.
    • S im -H int: 94.76.
    • SIGN 2000: 93.42.
    • Ironic intent: only 52.81. NBoW scores 61.72 and InferSent 11.72 on this test.
  • Negatives were made by hand. Automatic antonym substitution from lexicons often produced ungrammatical sentences.
  • RoBERTa's typical irony errors involve:
    • rhetorical questions;
    • pragmatic inference;
    • "[I wish] that" desideratives;
    • irony markers such as hashtags, stretched spellings ("grrrrreat") and embedded metaphors.

Interesting ideas

  • I1. Existing paraphrase corpora (figurative ↔ literal) can be recycled into entailment probes almost for free, with one-word antonym swaps as minimal-pair negatives that isolate a single phenomenon.
  • I2. Irony is split into two separate tests: recovering what the speaker meant, and merely detecting that the speaker was being ironic; models can pass one and fail the other badly.
  • I3. Classification labels can be turned into inference problems with a fixed template hypothesis ("X was ironic"), which tests whether a reader really derives the label from the text.
  • I4. Errors are sorted by a typology of rephrasing strategies and irony markers instead of being reported as one accuracy number. This shows that success on antonym and negation cases hides failures on pragmatic ones.
  • I5. Simile failures come from missing perceptual knowledge linking a vehicle to its implied property (tomato → red, cement slabs → motionless), and novel metaphors fail more often than entrenched ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): build negative-control suites by swapping one predicate for its antonym in silver pairs, and check that the answer flips while everything else in the world model stays the same.
  • I2 (frames): evaluate the irony module in prag separately for detection (a speech-act flag) and for inversion of the asserted proposition, so each failure is traced to the right step.
  • I3 (answer): recast annotated attributes of fables (sarcasm, deception, a character's intent) into templated yes/no questions for the readers.
  • I4 (tokens): normalize elongated spellings and split multiword hashtags while keeping a marker feature, so later stages can still use the signal of exaggeration.
  • I5 (world): add a vehicle-to-property table and a conventionality score for metaphoric verb senses to the compiled knowledge, so similes and metaphors resolve to explicit literal states.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🤝Educational Testing ServicePrinceton · United States14 works · 1,879 citations🎓Barnard CollegeNew York · United States3 works · 196 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Reasoning · 510 works Idioms & multiword expressions · 155 works

Cite

Cite as Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak and Smaranda Muresan. 2021. Figurative Language in Recognizing Textual Entailment. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.297

@inproceedings{chakrabarty2021figurative,
  title = {{Figurative Language in Recognizing Textual Entailment}},
  author = {Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak and Smaranda Muresan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.297},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.297}
}

References

50 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri. 2008. Metaphor in textual entailment. In Coling 2008: Companion volume: Posters, pages 3–6, Manchester, UK. Coling 2008 Organizing Committee. find
  2. Rodrigo Agerri, John Barnden, Mark Lee, and Alan Wallington. 2008. Textual entailment as an evaluation framework for metaphor resolution: A proposal. In Semantics in Text Processing. STEP 2008 Conference Proceedings, pages 357–363. College Publications. ↗
  3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, and Bernardo Magnini. 2006. The second pascal recognising textual entailment challenge. find
  4. Jean-Philippe Bernardy and Stergios Chatzikyriakidis. 2019. What kind of natural language inference are nlp systems learning: Is this enough? find
  5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. ↗
  6. Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6455–6469, Online. Association for Computational Linguistics. find
  7. Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021. MERMAID: Metaphor generation with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4250–4261, Online. Association for Computational Linguistics. ↗
  8. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  9. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. find
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
  11. I. Dasgupta, D. Guo, A. Stuhlmüller, S. J. Gershman, and N. D. Goodman. 2018. Evaluating Compositionality in Sentence Embeddings. ArXiv e-prints. ↗
  12. Shelly Dews, Joan Kaplan, and Ellen Winner. 2007. Why not say it directly? the social functions of irony. Irony in language and thought, pages 297–318. find
  13. Susan R Fussell and Mallie M Moss. 1998. Figurative language in emotional communication. Social and cognitive approaches to interpersonal communication, pages 113–141. find
  14. Omar Carlo Gioacchino Gelo and Erhard Mergenthaler. 2012. Unconventional metaphors and emotional-cognitive regulation in a metacognitive interpersonal therapy. Psychotherapy Research, 22(2):159–175. ↗
  15. Richard J Gerrig and Raymond W Gibbs Jr. 1988. Beyond the lexicon: Creativity in language production. Metaphor and Symbol, 3(3):1–19. ↗
  16. Debanjan Ghosh. 2018. An Empirical Study of Verbal Irony. Ph.D. thesis, Rutgers, The State University of New Jersey. ↗
  17. Debanjan Ghosh and Smaranda Muresan. 2018. “with 1 follower i must be awesome :p”. exploring the role of irony markers in irony recognition. Proceedings of ICWSM. our card
  18. Debanjan Ghosh, Elena Musi, and Smaranda Muresan. 2020. Interpreting verbal irony: Linguistic strategies and the connection to theType of semantic incongruity. In Proceedings of the Society for Computation in Linguistics 2020, pages 82–93, New York, New York. Association for Computational Linguistics. our card
  19. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. ↗ our card
  20. Sam Glucksberg. 1998. Understanding metaphors. Current Directions in Psychological Science, 7(2):39–43. find
  21. E Dario Gutierrez, Ekaterina Shutova, Tyler Marghetis, and Benjamin Bergen. 2016. Literal and metaphorical senses in compositional distributional semantic models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 183–193. find
  22. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  23. Julia Jorgensen. 1996. The functions of sarcastic irony in speech. Journal of pragmatics, 26(5):613–634. ↗
  24. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 235–249, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  25. Walter Kintsch and Anita R Bowles. 2002. Metaphor comprehension: What makes a metaphor difficult to understand? Metaphor and symbol, 17(4):249–262. ↗
  26. Thomas Kober, Sander Bijl de Vroe, and Mark Steedman. 2019. Temporal and aspectual entailment. In Proceedings of the 13th International Conference on Computational Semantics-Long Papers, pages 103– 119. find
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  28. James H Martin. 2006. A corpus-based analysis of context effects on metaphor comprehension. ↗
  29. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  30. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  31. Anthony M Paul, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 1970. Figurative language. In Philosophy & Rhetoric, pages 225–248. find
  32. Ellie Pavlick and Chris Callison-Burch. 2016. Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173. Association for Computational Linguistics. find
  33. Lotem Peled and Roi Reichart. 2017. Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1690–1700, Vancouver, Canada. Association for Computational Linguistics. our card
  34. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗ our card
  35. Adam Poliak. 2020. A survey on recognizing textual entailment as an nlp evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92–109. our card
  36. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 337–340, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  37. Kyle Richardson, Hai Na Hu, Lawrence S. Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In AAAI, volume abs/1909.07521. ↗ our card
  38. Richard M Roberts and Roger J Kreuz. 1994. Why do people use figurative language? Psychological science, 5(3):159–163. ↗
  39. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics. our card
  40. Ekaterina V Shutova. 2011. Computational approaches to figurative language. Technical report, University of Cambridge, Computer Laboratory. find
  41. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. Philosophy, 3:143–184. find
  42. Gerard Steen. 2010. A method for linguistic metaphor identification: From MIP to MIPVU, volume 14. John Benjamins Publishing. find
  43. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50. our card
  44. Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal reasoning in natural language inference. our card
  45. Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016. Metaphor: A computational perspective. Synthesis Lectures on Human Language Technologies, 9(1):1–160. ↗
  46. Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. Probing neural language models for human tacit assumptions. In 42nd Annual Virtual Meeting of the Cognitive Science Society (CogSci). ↗
  47. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005. our card
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  50. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In ACL. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakrabarty-2021-figurative-language-recognizing-textual-entailment/