Elamur Research⌕
☰
home / papers / 2024 / Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models

Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models

Paper · 2024 · Amelie Robrecht, Judith Sieker, Clara Lachenmaier et al.

Summary

This extended abstract from Bielefeld University (Robrecht, Sieker, Lachenmaier, Zarrieß, Kopp) argues that research on the pragmatic skills of LLMs is scattered and lacks clear definitions. Morphology and syntax have settled benchmarks, but pragmatics is a loose collection of context-dependent phenomena. Without a map of these phenomena, results are hard to compare or interpret. The authors propose dividing the field into two groups:

  • Discourse pragmatics covers presupposition, implicature and figurative language. These are tied to text coherence and do not need live interaction.
  • Interactional pragmatics covers politeness, turn-taking, repair and adaptation. These arise from the interplay between interlocutors.

They then sketch how prior LLM studies fall into these groups. Discourse phenomena have been probed fairly often, using instruments borrowed from psycholinguistics. Interactional phenomena are rarely tested, and the few studies that exist mostly concern Theory of Mind or perspective-taking. The paper reports no experiments. Its contribution is the proposed taxonomy plus a promised survey of phenomena, methods and models. This matters because claims about LLM reasoning and grounding often rest on pragmatic evidence.

Key points

  • The paper adopts Cummings (2013): pragmatics covers intentionally conveyed meaning beyond the literal content. It aligns discourse pragmatics with Korta and Perry's "near-side" and interactional pragmatics with "far-side".
  • It notes that turn-taking and repair are missing from the Stanford Encyclopedia overview, even though they are clearly pragmatic and relevant to grounding.
  • Surveyed discourse-pragmatics work includes:
    • implicature understanding in OPT, T5 and GPT-4 (Ruis et al., 2022);
    • scalar inferences in BERT compared against human data (Hu et al., 2023);
    • GPT-2 metaphor understanding explained via Rational Speech Act theory (Carenini et al., 2023);
    • seven phenomena tested across GPT-2, GPT-3 and T5.
  • Surveyed interactional-pragmatics work includes:
    • GPT-3 and GPT-4 deliberately lowering their apparent abilities to play other personas (Milička et al., 2024);
    • perspective-taking in GPT-3, GPT-4 and Llama2 with chain-of-thought prompting (Wilf et al., 2023).
  • Interactional pragmatics lacks instruments and measures because it is tied to spoken, spontaneous and adaptive interaction. Whether Theory of Mind belongs to it is left open.

Interesting ideas

  • I1. Splitting pragmatics into text-internal reasoning and between-speaker interaction gives separate scopes and separate test sets instead of one blurry category.
  • I2. A usable working definition treats pragmatic meaning as whatever the speaker intends beyond the literal content, which forces an explicit contrast between what is said and what is meant.
  • I3. Testing models with established psycholinguistic instruments and comparing them to human response patterns, as in the BERT scalar-inference study, grounds evaluation in known human behaviour.
  • I4. Explaining metaphor and scalar results through Rational Speech Act theory turns a score into a speaker–listener reasoning account.
  • I5. Flagging Theory of Mind and perspective-taking as borderline cases suggests keeping an agent's beliefs separate from what is actually true.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The prag crate could group implicature, presupposition, irony and metaphor as a discourse layer, and put politeness, repair and turn-taking markers in a separate dialogue layer with their own tests.
  • I2 (frames): Each implicature or irony output could store both the literal frame and the inferred extra proposition, keeping the inference step explainable.
  • I3 (learn): Pragmatics evaluation could reuse such classic test items with human baselines, so prag scores are judged against people, not only against an LLM teacher.
  • I4 (frames): A small RSA-style step could choose between literal and strengthened readings (e.g. "some" read as "not all") and record why one alternative was rejected.
  • I5 (world): The world model could hold per-character belief states apart from narrator-level facts, so fable QA can answer what a character thinks versus what happened.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bielefeld UniversityBielefeld · Germany35 works · 1,196 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksDiscourse analysis · 466 works

Cite

Cite as Amelie Robrecht, Judith Sieker, Clara Lachenmaier, Sina Zarieß and Stefan Kopp. 2024. Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models. arXiv:2408.03074.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.03074 · arXiv 2408.03074

@misc{robrecht2024analysis,
  title = {{Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models}},
  author = {Amelie Robrecht, Judith Sieker, Clara Lachenmaier, Sina Zariess and Stefan Kopp},
  year = {2024},
  doi = {10.48550/arxiv.2408.03074},
  eprint = {2408.03074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.03074}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mira Ariel. 2010. Defining Pragmatics, 1 edition. Cambridge University Press. find
  2. J Maxwell Atkinson and John Heritage. 1984. Structures of social action. Cambridge University Press. find
  3. Agnes Axelsson and Gabriel Skantze. 2023. Do You Follow?: A Fully Automated System for Adaptive Robot Presenters. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, pages 102–111, Stockholm, Sweden. ACM. find
  4. Yehoshua Bar-Hillel. 1971. Out of the pragmatic waste-basket. Linguistic Inquiry, 2(3):401–407. find
  5. Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press. find
  6. Gaia Carenini, Louis Bodot, Walter Schaeken, Luca Bischetti, and Valentina Bambini. 2023. Large Language Models Behave (Almost) As Rational Speech Actors: Insights From Metaphor Understanding. find
  7. Tyler A. Chang and Benjamin K. Bergen. 2023. Language Model Behavior: A Comprehensive Survey. arXiv preprint. ArXiv:2303.11504 [cs]. ↗
  8. Louise Cummings. 2013. Pragmatics: A multidisciplinary perspective. Routledge. find
  9. Judith Degen. 2023. The Rational Speech Act Framework. Annual Review of Linguistics, 9(1):519–540. ↗
  10. Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34–48. ↗
  11. Michael C. Frank and Noah D. Goodman. 2012. Predicting Pragmatic Reasoning in Language Games. Science, 336(6084):998–998. ↗
  12. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. Understanding Social Reasoning in Language Models with Language Models. arXiv preprint. ArXiv:2306.15448 [cs]. ↗
  13. Erving Goffman. 1955. On face-work: An analysis of ritual elements in social interaction. Psychiatry, 18(3):213–231. find
  14. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics. ↗
  15. Jennifer Hu, Roger Levy, Judith Degen, and Sebastian Schuster. 2023. Expectations over Unspoken Alternatives Predict Pragmatic Inferences. arXiv preprint. ArXiv:2304.04758 [cs]. ↗
  16. Malte F. Jung. 2017. Affective Grounding in Human-Robot Interaction. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, pages 263–273, Vienna Austria. ACM. find
  17. Kepa Korta and John Perry. 2020. Pragmatics. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Spring 2020 edition. Metaphysics Research Lab, Stanford University. find
  18. Shikhar Kumar, Eliran Itzhak, Yael Edan, Galit Nimrod, Vardit Sarne-Fleischmann, and Noam Tractinsky. 2022. Politeness in Human–Robot Interaction: A Multi-Experiment Study with Non-Humanoid Robots. International Journal of Social Robotics, 14(8):1805–1820. ↗
  19. Geoffrey N Leech. 2014. The pragmatics of politeness. Oxford University Press, USA. ↗
  20. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, page S1364661324000275. ↗
  21. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. ↗
  22. Jiří Milička, Anna Marklová, Klára VanSlambrouck, Eva Pospíšilová, Jana Šimsová, Samuel Harvan, and Ondřej Drobil. 2024. Large language models are able to downplay their cognitive abilities to fit the persona they simulate. PLOS ONE, 19(3):e0298522. ↗
  23. Amelie Sophie Robrecht, Markus Rothgänger, and Stefan Kopp. 2023. A Study on the Benefits and Drawbacks ofAdaptivity in AI-generated Explanations. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. find
  24. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs. Publisher: [object Object] Version Number: 2.
  25. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pages 7– 55. Elsevier. find
  26. Judith Sieker, Oliver Bott, Torgrim Solstad, and Sina Zarrieß. 2023. Beyond the Bias: Unveiling the Quality of Implicit Causality Prompt Continuations in Language Models. In Proceedings of the 16th International Natural Language Generation Conference, pages 206–220, Prague, Czechia. Association for Computational Linguistics. ↗
  27. Judith Sieker and Sina Zarrieß. 2023. When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 180–198, Singapore. Association for Computational Linguistics. ↗
  28. Gabriel Skantze. 2021. Turn-taking in Conversational Systems and Human-Robot Interaction: A Review. Computer Speech & Language, 67:101178. ↗
  29. Sonja Stange. 2022. Tell Me Why (and What)! Self-Explanations for Autonomous Social Robot Behavior. Ph.D. thesis, Universität Bielefeld. Artwork Size: 9237985 bytes Medium: application/pdf Publisher: Universität Bielefeld. find
  30. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. ↗
  31. Alex Wilf, Sihyun Shawn Lee, Paul Pu Liang, and Louis-Philippe Morency. 2023. Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities. arXiv preprint. ArXiv:2311.10227 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/robrecht-2024-towards-analysis-discourse-interactional-pragmatic/