Elamur Research⌕
☰
home / papers / 2025 / Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges

Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges

Summary

The survey examines how the pragmatic competence of language models is evaluated, that is, their ability to understand meaning that depends on context and does not follow from the literal content of words. The authors systematize resources for evaluating pragmatics in NLP and group datasets by the phenomena they test. For each group, they describe how tasks are formulated, how data are collected, which metrics and evaluation methods are used, and how suitable these are for real-world applications. A separate discussion addresses how LLMs can assist linguistic pragmatics itself: they can generate stimuli for experiments and produce draft annotations that are then verified by humans. The main conclusion is that, despite the progress of LLMs, their pragmatic abilities remain insufficiently tested. Existing benchmarks lack contextual richness, and their data are predominantly textual. The work is useful as a map of existing resources and a guide for building more precise benchmarks. The survey itself covers only English-language publications, so datasets in other languages, including Ukrainian, may not have been included.

Key points

  • Phenomena are sorted into five groups in the taxonomy: deixis and context; presupposition and implicature; recognition of intent along with speech acts; coherence and discourse; and social pragmatics (politeness, the notion of "face", and cultural and gender factors).
  • The covered resources include color reference tasks (Monroe et al., 2017, 2018) and scalar implicatures with graded adjectives (Nizamani et al., 2024). Also discussed are manner implicature in LLMs (Cong, 2024), implicatures and social biases in social media (Sap et al., 2020), and the discourse resources of Miltsakaki et al. (2004) and Prasad et al. (2008, 2018).
  • The survey proceeds from phenomena to NLP tasks, then to dataset construction methods, metrics, and evaluation techniques, and concludes with gaps and recommendations.
  • For experimental pragmatics, LLMs are proposed as stimulus generators and preliminary annotators, but the output must always be reviewed by a human. The authors also stress that NLP practitioners, cognitive scientists, and computational linguists should work together.
  • Limitations of the survey: only English-language literature from NLP conferences and predominantly textual data. To avoid anthropomorphization, the authors evaluate only the models' textual outputs and do not attribute intentions to them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.425 · arXiv 2502.12378

@inproceedings{ma2025pragmatics,
  title = {{Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges}},
  author = {Ma, Bolei and Li, Yuting and Zhou, Wei and Gong, Ziwei and Liu, Yang Janet and Jasinskaja, Katja and Friedrich, Annemarie and Hirschberg, Julia and Kreuter, Frauke and Plank, Barbara},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.425},
  eprint = {2502.12378},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.acl-long.425}
}

References

129 references, 51 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351. find our card
  2. Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC‘16), pages 2721–2727, Portorož, Slovenia. European Language Resources Association (ELRA). find our card
  3. Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
  4. John Langshaw Austin. 1975. How to do things with words. Harvard University Press. find
  5. Imran Sarwar Bajwa and Muhammad Abbas Choudhary. 2006. A rule based system for speech language context understanding. Journal of Donghua University (English Edition) Vol, 23(06). find
  6. Yuwei Bao, Sayan Ghosh, and Joyce Chai. 2022. Learning to mediate disparities towards pragmatic communication. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2829–2842, Dublin, Ireland. Association for Computational Linguistics. find
  7. Betty J Birner. 2012. Introduction to pragmatics. John Wiley & Sons. find
  8. Daniela Braga, Luís Coelho, João P. Teixeira, and Dia-mantino Freitas. 2006. Progmatica: A prosodic database for European Portuguese. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC‘06), Genoa, Italy. European Language Resources Association (ELRA). find
  9. Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage. 4. Cambridge University Press. find our card
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. find our card
  11. Dominique Brunato. 2025. Learning from impairment: Leveraging insights from clinical linguistics in language modelling research. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4167–4174, Abu Dhabi, UAE. Association for Computational Linguistics. find
  12. Sven Buechel and Udo Hahn. 2017. EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 578–585, Valencia, Spain. Association for Computational Linguistics. find
  13. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860–6868. find our card
  14. Erik Cambria. 2025. Pragmatics Processing, pages 229–338. Springer Nature Switzerland, Cham. find
  15. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3). find our card
  16. Yan Cong. 2024. Manner implicatures in large language models. Scientific Reports, 14(1):29113. find
  17. Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021. “I‘ll be there for you”: The one with understanding indirect answers. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 1–11, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. find our card
  18. Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. GuessWhat?! Visual Object Discovery through Multi-modal Dialogue . In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4466–4475, Los Alamitos, CA, USA. IEEE Computer Society. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  20. Esin Durmus, Faisal Ladhak, and Claire Cardie. 2019. The role of pragmatic and discourse context in determining argument impact. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5668–5678, Hong Kong, China. Association for Computational Linguistics. find our card
  21. Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling. find our card
  22. Penelope Eckert and Sally McConnell-Ginet. 1992. Think practically and look locally: Language and gender as community-based practice. Annual review of anthropology, pages 461–490. find
  23. Daniel Fried, Nicholas Tomlin, Jennifer Hu, Roma Patel, and Aida Nematzadeh. 2023. Pragmatics in language grounding: Phenomena, tasks, and modeling approaches. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12619– 12640, Singapore. Association for Computational Linguistics. find our card
  24. Annemarie Friedrich, Nianwen Xue, and Alexis Palmer. 2023. A kind introduction to lexical and grammatical aspect, with a survey of computational approaches. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 599–622, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  25. Elizabeth Jasmi George and Radhika Mamidi. 2020. Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316–2323. Third International Conference on Computing and Network Communications (Co-CoNet’19). find our card
  26. J.J. Godfrey, E.C. Holliman, and J. McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517– 520 vol.1. find
  27. Ziwei Gong, Muyin Yao, Xinyi Hu, Xiaoning Zhu, and Julia Hirschberg. 2024. A mapping on current classifying categories of emotions used in multimodal models for emotion recognition. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 19–28, St. Julians, Malta. Association for Computational Linguistics. find
  28. Andrew Greasley and Chris Owen. 2016. Behavior in models: A framework for representing human behavior. In Behavioral Operational Research: Theory, Methodology and Practice, pages 47–63, London. Palgrave Macmillan UK. find
  29. Claudio Greco, Diksha Bagade, Dieu-Thu Le, and Raffaella Bernardi. 2023. She adapts to her student: An expert pragmatic speaker tailoring her referring expressions to the layman listener. Frontiers in Artificial Intelligence, Volume 6 - 2023. find
  30. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Speech Acts: Syntax and Semantics Volume 3, pages 41–58. Academic Press, New York. ↗
  31. Crina Grosan, Ajith Abraham, Crina Grosan, and Ajith Abraham. 2011. Rule-based expert systems. Intelligent systems: A modern approach, pages 149–185. find
  32. Robert M. Groves. 2011. Three eras of survey research. Public Opinion Quarterly, 75(5):861–871. find
  33. Martin Guardado. 2018. What is discourse?, pages 70–78. De Gruyter Mouton, Berlin, Boston. find
  34. Jiaqi Guo, Ziliang Si, Yu Wang, Qian Liu, Ming Fan, Jian-Guang Lou, Zijiang Yang, and Ting Liu. 2021. Chase: A large-scale and pragmatic Chinese dataset for cross-database context-dependent text-to-SQL. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2316–2331, Online. Association for Computational Linguistics. find
  35. Mustafa Halat and Ümit Atlamaz. 2024. ImplicaTR: A granular dataset for natural language inference and pragmatic reasoning in Turkish. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), pages 29– 41, Bangkok, Thailand and Online. Association for Computational Linguistics. find
  36. Susan C Herring, Dieter Stein, and Tuija Virtanen. 2013. Introduction to the pragmatics of computer-mediated communication. Pragmatics of computer-mediated communication, pages 3–32. find
  37. Julia Hirschberg. 1985. A theory of scalar implicature. University of Pennsylvania. find
  38. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online. Association for Computational Linguistics. ↗ our card
  39. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4194–4213, Toronto, Canada. Association for Computational Linguistics. ↗ our card
  40. Yan Huang. 2017. Introduction: What is pragmatics? In The Oxford Handbook of Pragmatics. Oxford University Press. find
  41. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  42. Mark Johnson. 2009. How the statistical revolution changes (computational) linguistics. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pages 3–11, Athens, Greece. Association for Computational Linguistics. find
  43. Lalitha Kameswari, Dama Sravani, and Radhika Mamidi. 2020. Enhancing bias detection in political news using pragmatic presupposition. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 1–6, Online. Association for Computational Linguistics. find
  44. Christoph Kern, Stephanie Eckman, Jacob Beck, Rob Chew, Bolei Ma, and Frauke Kreuter. 2023. Annotation sensitivity: Training data collection methods affect model performance. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14874–14886, Singapore. Association for Computational Linguistics. find
  45. Fereshte Khani, Noah D. Goodman, and Percy Liang. 2018. Planning, inference and pragmatics in sequential language games. Transactions of the Association for Computational Linguistics, 6:543–555. find our card
  46. Yeeun Kim, Youngrok Choi, Eunkyung Choi, JinHwan Choi, Hai Jin Park, and Wonseok Hwang. 2024. Developing a pragmatic benchmark for assessing Korean legal language understanding in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5573–5595, Miami, Florida, USA. Association for Computational Linguistics. find our card
  47. Kana Koyano, Hitomi Yanaka, Koji Mineshima, and Daisuke Bekki. 2022. Annotating Japanese numeral expressions for a logical and pragmatic inference dataset. In Proceedings of the 18th Joint ACL -ISO Workshop on Interoperable Semantic Annotation within LREC2022, pages 127–132, Marseille, France. European Language Resources Association. find
  48. Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic grammatical error correction. In Proceedings of ArabicNLP 2023, pages 101–119, Singapore (Hybrid). Association for Computational Linguistics. find
  49. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223, Melbourne, Australia. Association for Computational Linguistics. find our card
  50. Bradford J. Lee and Cook R. Daniel. 2024. Exploring the potential of ai for pragmatics instruction. Technology in Language Teaching & Learning, 6(3):1521. find
  51. Stephen C Levinson. 1983. Pragmatics. Cambridge University Press. find
  52. Stephen C Levinson. 2006. Deixis. The handbook of pragmatics, pages 97–121. find
  53. Hengli Li, Song-Chun Zhu, and Zilong Zheng. 2023a. Diplomat: A dialogue dataset for situated pragmatic reasoning. In Advances in Neural Information Processing Systems, volume 36, pages 46856–46884. Curran Associates, Inc. find
  54. Jiaxuan Li, Minxi Yang, Dahua Gao, Wenlong Xu, and Guangming Shi. 2024. Pace: A pragmatic agent for enhancing communication efficiency using large language models. Preprint, arXiv:2402.01750. ↗
  55. Yiyuan Li, Rakesh Menon, Sayan Ghosh, and Shashank Srivastava. 2023b. Pragmatic reasoning unlocks quantifier semantics for foundation models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 573–591, Singapore. Association for Computational Linguistics. find
  56. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On LLMs-driven synthetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11065–11082, Bangkok, Thailand. Association for Computational Linguistics. find our card
  57. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “I‘d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425, Online. Association for Computational Linguistics. find our card
  58. Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael A. Hedderich, Barbara Plank, and Frauke Kreuter. 2024a. The potential and challenges of evaluating attitudes, opinions, and values in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8783–8805, Miami, Florida, USA. Association for Computational Linguistics. find
  59. Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xinpeng Wang, Markus Herklotz, Frauke Kreuter, Barbara Plank, and Matthias Assenmacher. 2024b. Algorithmic fidelity of large language models in generating synthetic german public opinions: A case study. Preprint, arXiv:2412.13169. ↗
  60. William C Mann. 1980. Toward a speech act theory for natural language processing. Information Sciences Institute, University of Southern California. find
  61. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. ↗ our card
  62. Andrea Martinenghi, Gregor Donabauer, Simona Amenta, Sathya Bursic, Mathyas Giudici, Udo Kruschwitz, Franca Garzotto, and Dimitri Ognibene. 2024. LLMs of catan: Exploring pragmatic capabilities of generative chatbots through prediction and classification of dialogue acts in boardgames’ multi-party dialogues. In Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024, pages 107–118, Torino, Italia. ELRA and ICCL. find
  63. Kleio-Isidora Mavridou, Annemarie Friedrich, Melissa Peate Sørensen, Alexis Palmer, and Manfred Pinkal. 2015. Linking discourse modes and situation entity types in a cross-linguistic corpus study. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 12–21, Lisbon, Portugal. Association for Computational Linguistics. find
  64. Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, and Min-Yen Kan. 2024. Discursive socratic questioning: Evaluating the faithfulness of language models’ understanding of discourse relations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6277–6295, Bangkok, Thailand. Association for Computational Linguistics. find our card
  65. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC‘04), Lisbon, Portugal. European Language Resources Association (ELRA). find our card
  66. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. ↗
  67. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. ↗ our card
  68. Will Monroe, Jennifer Hu, Andrew Jong, and Christopher Potts. 2018. Generating bilingual pragmatic color references. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2155–2165, New Orleans, Louisiana. Association for Computational Linguistics. find
  69. Christin Müller and Barbara Plank. 2024. Indirec-tQA: Understanding indirect answers to implicit polar questions in French and Spanish. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9025–9035, Torino, Italia. ELRA and ICCL. find
  70. Arbi Haza Nasution and Aytuğ Onan. 2024. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks. IEEE Access, 12:71876–71900. find
  71. Rashid Nizamani, Sebastian Schuster, and Vera Demberg. 2024. SIGA: A naturalistic NLI dataset of English scalar implicatures with gradable adjectives. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14784–14795, Torino, Italia. ELRA and ICCL. find our card
  72. Anais Ollagnier. 2024. CyberAgressionAdo-v2: Leveraging pragmatic-level information to decipher online hate in French multiparty chats. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4287–4298, Torino, Italia. ELRA and ICCL. find
  73. Juri Opitz, Shira Wein, and Nathan Schneider. 2025. Natural language processing relies on linguistics. Computational Linguistics, pages 1–24. find
  74. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–41, Los Angeles. Association for Computational Linguistics. find our card
  75. Lalchand Pandia, Yan Cong, and Allyson Ettinger. 2021. Pragmatic competence of pre-trained language models through the lens of discourse connectives. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 367–379, Online. Association for Computational Linguistics. ↗ our card
  76. Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, and Sungeun Lee. 2024a. Pragmatic competence evaluation of large language models for the korean language. Preprint, arXiv:2403.12675. ↗ our card
  77. Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong, Youngeun Koo, Soonha Hwang, Seonwoo Park, and Sungeun Lee. 2024b. MultiPragEval: Multilingual pragmatic evaluation of large language models. In Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP, pages 96–119, Miami, Florida, USA. Association for Computational Linguistics. find our card
  78. Paolo Pedinotti, Emmanuele Chersoni, Enrico Santus, and Alessandro Lenci. 2022. Pragmatic and logical inferences in NLI systems: The case of conjunction buttressing. In Proceedings of the Second Workshop on Understanding Implicit and Underspecified Language, pages 8–16, Seattle, USA. Association for Computational Linguistics. find our card
  79. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗ our card
  80. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC‘08), Marrakech, Morocco. European Language Resources Association (ELRA). find our card
  81. Rashmi Prasad, Bonnie Webber, and Alan Lee. 2018. Discourse annotation in the PDTB: The next generation. In Proceedings of the 14th Joint ACL-ISO Workshop on Interoperable Semantic Annotation, pages 87–97, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  82. Peng Qi, Nina Du, Christopher Manning, and Jing Huang. 2023. PragmatiCQA: A dataset for pragmatic question answering in conversations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6175–6191, Toronto, Canada. Association for Computational Linguistics. find our card
  83. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics. find
  84. Ines Reinig, Ines Rehbein, and Simone Paolo Ponzetto. 2024. How to do politics with words: Investigating speech acts in parliamentary debates. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8287– 8300, Torino, Italia. ELRA and ICCL. find
  85. Egil Rønningstad, Erik Velldal, and Lilja Øvrelid. 2024. A GPT among annotators: LLM-based entity-level sentiment annotation. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 133–139, St. Julians, Malta. Association for Computational Linguistics. find
  86. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15295–15311, Bangkok, Thailand. Association for Computational Linguistics. find our card
  87. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms. In Advances in Neural Information Processing Systems, volume 36, pages 20827–20905. Curran Associates, Inc. find our card
  88. Emily Sadlier-Brown, Millie Lou, Miikka Silfverberg, and Carla Kam. 2024. How useful is context, actually? comparing LLMs and humans on discourse marker prediction. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 231–241, Bangkok, Thailand. Association for Computational Linguistics. find
  89. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find our card
  90. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463– 4473, Hong Kong, China. Association for Computational Linguistics. find
  91. Uli Sauerland and Petra B Schumacher. 2025. Pragmatics: two decades of theory and experiment growing together. Aktuelle Entwicklungen in der linguistischen Forschung, 35:247. find
  92. Ayse Pinar Saygin and Ilyas Cicekli. 2002. Pragmatics in human-computer conversations. Journal of Pragmatics, 34(3):227–258. ↗
  93. Florian Schwarz. 2017. Experimental pragmatics. In Oxford Research Encyclopedia of Linguistics. Oxford University Press. find
  94. Ron Scollon, Suzanne Wong Scollon, and Rodney H Jones. 2011. Intercultural communication: A discourse approach. John Wiley & Sons. find
  95. John R. Searle. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press. find
  96. Omar Shaikh, Caleb Ziems, William Held, Aryan Pari-ani, Fred Morstatter, and Diyi Yang. 2023. Modeling cross-cultural pragmatic inference with codenames duet. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6550–6569, Toronto, Canada. Association for Computational Linguistics. find our card
  97. Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model alignment: A survey. Preprint, arXiv:2309.15025. ↗ our card
  98. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don‘t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5263–5281, Mexico City, Mexico. Association for Computational Linguistics. find
  99. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics. find our card
  100. Damien Sileo, Philippe Muller, Tim Van de Cruys, and Camille Pradel. 2022. A pragmatics-centered evaluation framework for natural language understanding. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2382–2394, Marseille, France. European Language Resources Association. find our card
  101. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Harvard University Press Cambridge, MA. find
  102. Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. 2024. PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities. In Findings of the Association for Computational Linguistics ACL 2024, pages 12075–12097, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. find our card
  103. Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd-Graber. 2024. Pregnant questions: The importance of pragmatic awareness in maternal health question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7253–7268, Mexico City, Mexico. Association for Computational Linguistics. find our card
  104. Robert Stalnaker, Milton K Munitz, and Peter Unger. 1977. Pragmatic presuppositions. In Proceedings of the Texas conference on performatives, presuppositions, and implicatures. Arlington, VA: Center for Applied Linguistics, pages 135–148. ERIC. find
  105. Andreea Stapleton. 2017. Deixis in modern linguistics. Essex Student Journal, 9. find
  106. Shane Storks, Qiaozi Gao, and Joyce Y. Chai. 2020. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. Preprint, arXiv:1904.01172. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1277–1285, Toronto, Canada. Association for Computational Linguistics. ↗ our card
  107. Saujas Vaduguru, Daniel Fried, and Yewen Pu. 2024. Generating pragmatic examples to train neural program synthesizers. In The Twelfth International Conference on Learning Representations. find
  108. Bram van Dijk, Tom Kouwenhoven, Marco Spruit, and Max Johannes van Duijn. 2023. Large language models: The need for nuance in current debates and a pragmatic perspective on understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12641–12654, Singapore. Association for Computational Linguistics. find
  109. Emiel van Miltenburg. 2017. Pragmatic descriptions of perceptual stimuli. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 1–10, Valencia, Spain. Association for Computational Linguistics. find
  110. Emiel van Miltenburg, Roser Morante, and Desmond Elliott. 2016. Pragmatic factors in image description: The case of negations. In Proceedings of the 5th Workshop on Vision and Language, pages 54– 59, Berlin, Germany. Association for Computational Linguistics. find
  111. Ece Takmaz, Nicolo’ Brandizzi, Mario Giulianelli, Sandro Pezzelle, and Raquel Fernandez. 2023. Speaking the language of your listener: Audience-aware adaptation via plug-and-play theory of mind. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4198–4217, Toronto, Canada. Association for Computational Linguistics. find
  112. Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang. 2020. DuSQL: A large-scale and pragmatic Chinese text-to-SQL dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6923–6935, Online. Association for Computational Linguistics. find our card
  113. Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, and Raquel Fernández. 2020. Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4350–4368, Online. Association for Computational Linguistics. find
  114. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery. find
  115. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 930–957, Miami, Florida, USA. Association for Computational Linguistics. find our card
  116. Deborah Tannen. 1990. You Just Don’t Understand: Women and Men in Conversation. New York: Ballan-tine Books. find
  117. Polina Tsvilodub and Michael Franke. 2023. Evaluating pragmatic abilities of image captioners on A3DS. find
  118. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. ↗ our card
  119. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1118–1127, Marseille, France. European Language Resources Association. find
  120. Hongyi Wu, Xinshu Shen, Man Lan, Shaoguang Mao, Xiaopeng Bai, and Yuanbin Wu. 2023. A multi-task dataset for assessing discourse coherence in Chinese essays: Structure, theme, and logic analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6673– 6688, Singapore. Association for Computational Linguistics. find our card
  121. Shengguang Wu, Shusheng Yang, Zhenglun Chen, and Qi Su. 2024. Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22583–22599, Miami, Florida, USA. Association for Computational Linguistics. find our card
  122. Mingyou Xiang, Mian Jia, and Xiaohui Bu. 2024. Introduction to Pragmatics, volume 9. Springer Nature. find
  123. Christine Yang, Duanchen Liu, Qingyun Yang, Zoey Liu, and Emily Prud’hommeaux. 2021. Predicting pragmatic discourse features in the language of adults with autism spectrum disorder. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 284–291, Online. Association for Computational Linguistics. find
  124. Akhila Yerukola, Saujas Vaduguru, Daniel Fried, and Maarten Sap. 2024. Is the pope catholic? yes, the pope is catholic. generative evaluation of non-literal intent resolution in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 265–275, Bangkok, Thailand. Association for Computational Linguistics. ↗
  125. George Yule. 1996. Pragmatics. Oxford university press. find
  126. Deniz Zeyrek, Amália Mendes, and Murathan Kurfalı. 2018. Multilingual extension of PDTB-style annotation: The case of TED multilingual discourse bank. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗
  127. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics. find our card
  128. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A grammar-based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074–2085, Online. Association for Computational Linguistics. ↗ our card
  129. Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. 2024. Is this the real life? is this just fantasy? the misleading success of simulating social interactions with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21692–21714, Miami, Florida, USA. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2025-pragmatics-era-large-language-models/