Elamur Research
☰
home / papers / 2023 / A fine-grained comparison of pragmatic language understanding in humans and language models

A fine-grained comparison of pragmatic language understanding in humans and language models

What it is about

Seven pragmatic phenomena (white lies, indirect requests, irony, flouted maxims, metaphor, humour, coherence) are tested with expert-written multiple-choice items: a 1–3 sentence story, a character's utterance, and «what did they mean?». The options are typed: the pragmatic answer, the literal answer, and distractors built from lexical overlap with the story, loose association, or a non-sequitur. Models up to 11B parameters and GPT-3.5 are compared with 374 human participants, zero-shot. Small models are at chance; the largest reach near-human accuracy on several tasks and, like humans, err mostly towards the literal option rather than the distractors; humour, irony and maxims (expectation violations) stay hardest. Removing the context story hurts models and humans on the same tasks, scrambling words barely hurts the models (a sign of lexical shortcuts), and item-level accuracy correlates with humans for indirect speech, irony, maxims and metaphor.

Ideas

For Mova

languages/en/34-irony-and-figurative-language.md (irony, metaphor, humour), 33-implicature-and-presupposition.md §2 (maxims), 31-speech-acts.md §5. Code: world/src/knowledge.rs (diagnosis by distractor type, perturbations), prag/src/irony.rs, prag/src/infer.rs. Measures: share of errors per distractor type; accuracy drop under context removal and scrambling.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-long.230.pdf

Summary

The paper asks whether modern language models grasp non-literal meaning the way people do, and how their failures differ from human ones. Earlier evaluations mostly scored answers as simply pragmatic or not. Here the authors use an expert-built English multiple-choice set from Floyd et al. covering seven phenomena. In each item, a short story and a character's utterance are followed by answer options that encode different interpretation strategies: the intended reading, a literal reading, and distractors built on shallow heuristics such as word overlap. GPT-2, Tk-Instruct, Flan-T5 and InstructGPT were tested with zero-shot prompting and compared with human participants on accuracy, error types and sensitivity to linguistic cues. The biggest models reach high accuracy and resemble humans in their choices. When they miss, they usually pick the literal option rather than a heuristic distractor. Models and people also appear to rely on similar cues. This suggests some pragmatic behavior can arise without explicit mental-state representations, though models remain weak where social expectations are violated.

Key points

  • The seven tasks are Deceits, Indirect speech, Irony, Maxims, Metaphor, Humor and Coherence, each with 20–40 hand-curated items. Code and data are at github.com/jennhu/lm-pragmatics.
  • Flan-T5 (XL) and text-davinci-002 performed best and most closely tracked the distribution of human responses.
  • Model errors leaned toward literal interpretations rather than heuristic-based distractors, mirroring human error patterns.
  • Performance was poor on Humor, Irony and Maxims, which points to trouble with social conventions.
  • The study has three main limitations:
    • It covers English only, with partly Western-centric norms.
    • The item set is small.
    • Apart from the OpenAI API models, only models of ≤11B parameters were tested.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.230 · arXiv 2212.06801

@inproceedings{hu2023fine,
  title = {{A fine-grained comparison of pragmatic language understanding in humans and language models}},
  author = {Hu, Jennifer and Floyd, Sammy and Jouravlev, Olessia and Fedorenko, Evelina and Gibson, Edward},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.230},
  eprint = {2212.06801},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.230}
}

References

99 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Felix K. Ameka and Marina Terkourafi. 2019. What if. . . ? Imagining non-Western perspectives on pragmatic theory and practice. Journal of Pragmatics, 145:72–82. find
  2. Jacob Andreas. 2022. Language Models as Agent Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769–5779, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  3. Clara Andrés-Roqueta and Napoleon Katsos. 2017. The Contribution of Grammar, Vocabulary and Theory of Mind in Pragmatic Language Competence in Children with Autistic Spectrum Disorders. Frontiers in Psychology, 8. ↗
  4. Ian Apperly. 2011. Mindreaders: The cognitive basis of "Theory of Mind". Psychology Press, New York. find
  5. Salvatore Attardo. 2000. Irony as relevant inappropriateness. Journal of Pragmatics, 32(6):793–826. ↗
  6. John L. Austin. 1975. How to do things with words. find
  7. Simge Aykan and Erhan Nalçacı. 2018. Assessing Theory of Mind by Humor: The Humor Comprehension and Appreciation Test (ToM-HCAT). Frontiers in Psychology, 9. ↗
  8. Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. 1985. Does the autistic child have a “theory of mind”? Cognition, 21(1):37–46. find
  9. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics. ↗ our card
  10. Leon Bergen, Roger Levy, and Noah D. Goodman. 2016. Pragmatic reasoning through semantic inference. Semantics and Pragmatics, 9. ↗
  11. Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021. Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4164–4173, Online. Association for Computational Linguistics. ↗
  12. Luca Bischetti, Irene Ceccato, Serena Lecce, Elena Cav-allini, and Valentina Bambini. 2019. Pragmatics and theory of mind in older adults’ humor comprehension. Current Psychology. ↗
  13. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience Grounds Language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718–8735, Online. Association for Computational Linguistics. ↗ our card
  14. Damián E. Blasi, Joseph Henrich, Evangelia Adamou, David Kemmerer, and Asifa Majid. 2022. Over-reliance on English hinders cognitive science. Trends in Cognitive Sciences, 26(12):1153–1170. Publisher: Elsevier. ↗
  15. W.C. Booth. 1974. A Rhetoric of Irony. Literature/Criticism - The University of Chicago Press. University of Chicago Press. find
  16. Susan E. Brennan, Alexia Galati, and Anna K. Kuhlen. 2010. Two Minds, One Dialog: Coordinating Speaking and Understanding. In Brian H. Ross, editor, Psychology of Learning and Motivation, volume 53, pages 301–344. Academic Press. find
  17. Penelope Brown and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Cambridge University Press. ↗ our card
  18. Penelope Brown, Mark A. Sicoli, and Olivier Le Guen. 2021. Cross-speaker repetition and epistemic stance in Tzeltal, Yucatec, and Zapotec conversations. Journal of Pragmatics, 183:256–272. ↗
  19. Stephen A. Butterfill and Ian A. Apperly. 2013. How to Construct a Minimal Theory of Mind. Mind & Language, 28(5):606–637. Publisher: John Wiley & Sons, Ltd. ↗
  20. Carl Camden, Michael T. Motley, and Ann Wilson. 1984. White lies in interpersonal communication: A taxonomy and preliminary investigation of social motivations. Western Journal of Speech Communication, 48(4):309–325. Publisher: Routledge. ↗
  21. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With Little Power Comes Great Responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263–9274, Online. Association for Computational Linguistics. ↗
  22. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin-odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv preprint. find
  23. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling Instruction-Finetuned Language Models. arXiv preprint. find
  24. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating Compositionality in Sentence Embeddings. In Proceedings of the Cognitive Science Society. find
  25. Lambert Deckers and Philip Kizer. 1975. Humor and the Incongruity Hypothesis. The Journal of Psychology, 90(2):215–218. ↗
  26. Judith Degen. 2023. The Rational Speech Act Framework. Annual Review of Linguistics, 9(1):519–540. ↗
  27. Marie-Julie Demedardi, Claire Brechet, Edouard Gen-taz, and Catherine Monnier. 2021. Prosocial lying in children between 4 and 11 years of age: The role of emotional understanding and empathy. Journal of Experimental Child Psychology, 203:105045. ↗
  28. Christina Dideriksen, Morten H Christiansen, Mark Dingemanse, Malte Højmark-Bertelsen, Christer Johansson, Kristian Tylén, and Riccardo Fusaroli. 2022. Language specific constraints on conversation: Evidence from Danish and Norwegian. PsyArXiv preprint. ↗
  29. Judit Dombi, Tetyana Sydorenko, and Veronika Timpe-Laughlin. 2022. Common ground, cooperation, and recipient design in human-computer interactions. Journal of Pragmatics, 193:4–20. ↗
  30. Ivan Enrici, Bruno G. Bara, and Mauro Adenzato. 2019. Theory of Mind, pragmatics and the brain: Converging evidence for the role of intention processing as a core feature of human communication. Pragmatics & Cognition, 26(1):5–38. Sammy Floyd, Olessia Jouravlev, Zachary Mineroff, Leon Bergen, Evelina Fedorenko, and Edward Gibson. In prep. Deciphering the structure of pragmatics: A large-scale individual differences investigation. ↗
  31. Simeon Floyd. 2021. Conversation and Culture. Annual Review of Anthropology, 50(1):219–240. Publisher: Annual Reviews. ↗
  32. Michael C Frank, Andrés Goméz Emilsson, Benjamin Peloquin, Noah D. Goodman, and Christopher Potts. 2018. Rational speech act models of pragmatic reasoning in reference games. PsyArXiv preprint. find
  33. Michael C. Frank and Noah D. Goodman. 2012. Predicting Pragmatic Reasoning in Language Games. Science, 336(6084):998–998. ↗ our card
  34. Michael Franke and Gerhard Jäger. 2016. Probabilistic pragmatics, or why Bayes’ rule is probably important for pragmatics. Zeitschrift für Sprachwissenschaft, 35(1):3–44. ↗
  35. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  36. Raymond W. Gibbs. 1979. Contextual effects in understanding indirect requests. Discourse Processes, 2(1):1–10. Publisher: Routledge. ↗
  37. Herbert P. Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Speech Acts, volume 3, pages 41–58. Academic Press. ↗
  38. Yuling Gu, Yao Fu, Valentina Pyatkin, Ian Magnus-son, Bhavana Dalvi Mishra, and Peter Clark. 2022. Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 84–93, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  39. Francesca G.E. Happé. 1993. Communicative competence and theory of mind in autism: A test of relevance theory. Cognition, 48(2):101–119. ↗
  40. Cecilia Heyes. 2014. Submentalizing: I Am Not Really Reading Your Mind. Perspectives on Psychological Science, 9(2):131–143. ↗
  41. Laurence R. Horn. 1972. On the semantic properties of logical operators in English. PhD Thesis, University of California Los Angeles. find
  42. Yik Kwan Hsu and Him Cheung. 2013. Two mentalizing capacities and the understanding of two types of lie telling in children. Developmental Psychology, 49:1650–1659. ↗
  43. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics. ↗
  44. Nir Jacoby and Evelina Fedorenko. 2020. Discourse-level comprehension engages medial frontal Theory of Mind brain regions even for expository texts. Language, Cognition and Neuroscience, 35(6):780–796. find
  45. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  46. Justine T. Kao, Leon Bergen, and Noah D. Goodman. 2014. Formalizing the Pragmatics of Metaphor Understanding. In Proceedings of the 36th Annual Meeting of the Cognitive Science Society. find
  47. Justine T. Kao and Noah D. Goodman. 2014. Let’s talk (ironically) about the weather: Modeling verbal irony. In Proceedings of the 36th Annual Meeting of the Cognitive Science Society. find
  48. Melissa Kline Struhl, Jeanne Gallée, Zuzanna Balewski, and Evelina Fedorenko. 2018. Understanding jokes draws most heavily on the Theory of Mind brain network. PsyArXiv preprint. find
  49. Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. arXiv preprint. find
  50. Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts. 2022. Concadia: Towards image-based text generation with a purpose. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4667–4684, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  51. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗ our card
  52. G. Lakoff and M. Johnson. 1980. Metaphors We Live By. University of Chicago Press. find
  53. Julian Michael. 2020. To Dissect an Octopus: Making Sense of the Form/Meaning Debate. find
  54. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the Evaluation of Theory of Mind through Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877, Hong Kong, China. Association for Computational Linguistics. ↗
  55. Jarrod Moss and Christian D. Schunn. 2015. Comprehension through explanation as the interaction of the brain’s coherence and cognitive control networks. Frontiers in Human Neuroscience, 9. ↗
  56. Alan M. Leslie, Ori Friedman, and Tim P. German. 2004. Core mechanisms in ‘theory of mind’. Trends in Cognitive Sciences, 8(12):528–533. ↗
  57. Stephen Levinson. 2000. Presumptive meaning: The theory of generalized conversational implicature. MIT Press. find
  58. Elissa Li, Sebastian Schuster, and Judith Degen. 2021. Predicting Scalar Inferences From "Or" to "Not Both" Using Neural Sentence Encoders. In Proceedings of the Society for Computation in Linguistics, volume 4. find our card
  59. Jin Li. 2012. Cultural Foundations of Learning: East and West. Cambridge University Press, Cambridge. ↗
  60. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. Place: Cambridge, MA Publisher: MIT Press. ↗ our card
  61. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the Ability of Language Models to Interpret Figurative Language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452, Seattle, United States. Association for Computational Linguistics. ↗
  62. Eleonore Lumer and Hendrik Buschmeier. 2022. Modeling Social Influences on Indirectness in a Rational Speech Act Approach to Politeness. In Proceedings of the 44th Annual Conference of the Cognitive Science Society. find
  63. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: A cognitive perspective. arXiv preprint. ↗
  64. R.A. Martin and T. Ford. 2018. The Psychology of Humor: An Integrative Approach. Academic Press. find
  65. Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Evaluating Theory of Mind in Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392–2400, Brussels, Belgium. Association for Computational Linguistics. ↗
  66. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. Analyzing Compositionality-Sensitivity of NLI Models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6867–6874. ↗
  67. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ↗
  68. Christopher Potts. 2020. Is it possible for language models to achieve language understanding? find
  69. Christopher Potts, Daniel Lassiter, Roger Levy, and Michael C. Frank. 2016. Embedded Implicatures as Pragmatic Inferences under Compositional Lexical Uncertainty. Journal of Semantics, 33(4):755–802. ↗
  70. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  71. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67. find
  72. Paula Rubio-Fernandez. 2021. Pragmatic markers: the missing link between language and Theory of Mind. Synthese, 199(1):1125–1158. ↗
  73. Paula Rubio-Fernandez and Julian Jara-Ettinger. 2020. Incrementality and efficiency shape pragmatics across languages. Proceedings of the National Academy of Sciences, 117(24):13399–13404. ↗
  74. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. Large language models are not zero-shot communicators. arXiv preprint. find
  75. Michael Henry Tessler and Michael Franke. 2018. Not unreasonable: Carving vague dimensions with contraries and contradictions. In Proceedings of the 40th Annual Conference of the Cognitive Science Society. find
  76. Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  77. Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis. 2021. Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4673–4686, Online. Association for Computational Linguistics. ↗
  78. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463– 4473, Hong Kong, China. Association for Computational Linguistics. ↗
  79. Anna Trosborg, editor. 2010. Pragmatics across Languages and Cultures. De Gruyter Mouton. ↗
  80. Ayse Pinar Saygin and Ilyas Cicekli. 2002. Pragmatics in human-computer conversations. Journal of Pragmatics, 34(3):227–258. ↗
  81. Corrie Vendetti, Deepthi Kamawar, and Katherine E. Andrews. 2019. Theory of mind and preschoolers’ understanding of misdeed and politeness lies. Developmental Psychology, 55(4):823–834. ↗
  82. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. ↗ our card
  83. Yizhong Wang, Swaroop Mishra, Pegah Alipoormo-labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  84. John R. Searle. 1975. Indirect Speech Acts. In Speech Acts, pages 59–82. Brill, Leiden, The Netherlands. ↗
  85. Chirag Shah and Emily M. Bender. 2022. Situating Search. In ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR ’22, pages 221–232, New York, NY, USA. Association for Computing Machinery. Event-place: Regensburg, Germany. find
  86. Nicola Spotorno, Eric Koun, Jérôme Prado, Jean-Baptiste Van Der Henst, and Ira A. Noveck. 2012. Neural evidence that utterance-processing entails mentalizing: The case of irony. NeuroImage, 63(1):25–39. ↗
  87. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI Models’ Performance on Figurative Language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  88. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A Large Language Model for Science. arXiv preprint. find our card
  89. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. arXiv preprint. find
  90. Thomas C. Veatch. 1998. A theory of humor. Humor, 11(2):161–216. find
  91. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations. find
  92. D. Wilson and D. Sperber. 2012. Meaning and Relevance. Cambridge University Press. find
  93. Deirdre Wilson and Dan Sperber. 1992. On verbal irony. Lingua, 87(1):53–76. find
  94. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  95. Erica J. Yoon, Michael Henry Tessler, Noah D. Goodman, and Michael C. Frank. 2016. Talking with tact: Polite language as a balance between informativity and kindness. In Proceedings of the Annual Meeting of the Cognitive Science Society. find
  96. Erica J. Yoon, Michael Henry Tessler, Noah D. Goodman, and Michael C. Frank. 2020. Polite Speech Emerges From Competing Social Goals. Open Mind, 4:71–87. ↗
  97. George Yule. 1996. Pragmatics, 1 edition. Oxford Introduction to Language Study. Oxford University Press. find
  98. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8799–8809. ↗
  99. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074–2085, Online. Association for Computational Linguistics. ↗ our card