Elamur Research⌕
☰
home / papers / 2026 / DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

Summary

The authors investigate how well large language models recognize conversational implicature in questions, that is, meaning that the speaker only hints at rather than states directly. They created the DRInQ benchmark. In it, the same question is presented in several different situations, and in each one it carries a different implicit meaning. For each context, the model selects the correct interpretation from five options. The remaining options are plausible readings of the same question in other situations. The data were collected semi-automatically: an LLM generates contexts from intent labels drawn from speech act theory, and humans verify the output. The authors tested 12 models and found a gap between generation and inference. Strong models can write convincing pragmatic scenarios, but they often fail to recover the intended implicature and choose overly strong interpretations that the context does not support. For smaller models, structured prompting narrows the gap with larger ones. A comparative writing study showed that humans write more cautious and more typical contexts, while models write more diverse ones whose implicatures are overly explicit. The work shows that evaluating pragmatics requires context-sensitive benchmarks.

Key points

  • DRInQ (Dataset for Recovering Implicature in Questions) consists of "question — context — interpretation" triples with five answer options. The dataset is openly released on GitHub (hjarai/drinq).
  • Context generation is guided by 23 intent labels grouped into four speech act categories: Directive, Assertive, Commissive, Expressive.
  • On the human-verified hard subset, the best model, GPT-4o, reaches 76% accuracy. The average annotator score is 89%.
  • 78 participants were recruited via Prolific for the verification and writing tasks. A consensus of at least 4 out of 5 annotators was reached for only 80.8% of the examples.
  • The resource is in English and reflects Anglo-Saxon social norms, so the authors do not expect it to transfer well to other cultural contexts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Implicature

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1597 · arXiv 2605.24267

@inproceedings{arai2026drinq,
  title = {{DRInQ: Evaluating Conversational Implicature with Controlled Context Variation}},
  author = {Arai, Hirona Jacqueline and Ren, Xiang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1597},
  eprint = {2605.24267},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1597}
}

References

54 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J.L. Austin, J.O. Urmson, and M. Sbisà. 1975. How to Do Things with Words: Second Edition. Harvard paperback. Harvard University Press. ↗
  2. Zouheir Banou, Sanaa El Filali, El Habib Benlahmar, Fatima-Zahra Alaoui, Laila El Jiani, and Hasnae Sakhi. 2025. A systematic review of figurative language detection: Methods, challenges, and multilingual perspectives. Natural Language Processing Journal, 13:100192. ↗
  3. Chiara Barattieri di San Pietro, Federico Frau, Veronica Mangiaterra, and Valentina Bambini. 2023. The pragmatic profile of chatgpt: Assessing the communicative skills of a conversational agent. Sistemi Intelligenti, XXXV:379–400. ↗
  4. P. Brown and S.C. Levinson. 1987. Politeness: Some Universals in Language Usage. Politeness: Some Universals in Language Usage. Cambridge University Press. find our card
  5. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  6. Herbert Paul Grice. 1989. Studies in the Way of Words. Harvard University Press, Cambridge. find
  7. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4194–4213, Toronto, Canada. Association for Computational Linguistics. ↗ our card
  8. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  9. Lea Krause and Piek T.J.M. Vossen. 2024. The Gricean maxims in NLP - a survey. In Proceedings of the 17th International Natural Language Generation Conference, pages 470–485, Tokyo, Japan. Association for Computational Linguistics. find our card
  10. Shreyas Kulkarni, Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2024. A report on the FigLang 2024 shared task on multimodal figurative language. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 115–119, Mexico City, Mexico (Hybrid). Association for Computational Linguistics. find
  11. Joshua Lee, Wyatt Fong, Alexander Le, Sur Shah, Kevin Han, and Kevin Zhu. 2025. Pragmatic metacognitive prompting improves llm performance on sarcasm detection. In Proceedings of the 1st Workshop on Computational Humor (CHum), pages 63–70. ↗
  12. Stephen C Levinson. 2000. Presumptive meanings. Language, Speech, and Communication. Bradford Books, Cambridge, MA. find
  13. Huihan Li, Liwei Jiang, Jena D. Huang, Hyunwoo Kim, Sebastin Santy, Taylor Sorensen, Bill Yuchen Lin, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024. Culture-gen: Revealing global cultural perception in language models through natural language prompting. CoRR, abs/2404.10199. ↗
  14. Tyler A. Chang and Benjamin K. Bergen. 2023. Language model behavior: A comprehensive survey. Computational Linguistics, 50:293–350. ↗
  15. Ryan Liu, Jiayi Geng, {Addison J.} Wu, Ilia Sucholut-sky, Tania Lombrozo, and {Thomas L.} Griffiths. 2025. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. Proceedings of Machine Learning Research, 267:38489–38517. Publisher Copyright: © 2025, by the authors.; 42nd International Conference on Machine Learning, ICML 2025 ; Conference date: 13-07-2025 Through 19-07-2025. find
  16. Elizabeth Jasmi George and Radhika Mamidi. 2020. Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316–2323. ↗ our card
  17. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “i’d rather just go to bed”: Understanding indirect answers. In Conference on Empirical Methods in Natural Language Processing. find our card
  18. Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, and Barbara Plank. 2025. Pragmatics in the era of large language models: A survey on datasets, evaluation, opportunities and challenges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8679–8696, Vienna, Austria. Association for Computational Linguistics. find our card
  19. Julia Mendelsohn, Ronan Le Bras, Yejin Choi, and Maarten Sap. 2023. From dogwhistles to bullhorns: Unveiling coded rhetoric with language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15162–15180, Toronto, Canada. Association for Computational Linguistics. find
  20. Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, and Min-Yen Kan. 2024. Discursive socratic questioning: Evaluating the faithfulness of language models’ understanding of discourse relations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6277–6295, Bangkok, Thailand. Association for Computational Linguistics. find our card
  21. Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth M. Daly, David Piorkowski, and John T. Richards. 2024. Language models in dialogue: Conversational maxims for human-AI interactions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14420–14437, Miami, Florida, USA. Association for Computational Linguistics. find
  22. OpenAI. 2024. Gpt-4o system card. find our card
  23. Seoyoon Park, Hyeji Choi, Minseon Kim, Subin An, Xiaonan Wang, Gyuri Choi, and Hansaem Kim. 2025. FLUID QA: A multilingual benchmark for figurative language usage in dialogue across English, Chinese, and Korean. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30268–30282, Suzhou, China. Association for Computational Linguistics. find
  24. Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning. ↗ our card
  25. Linlu Qiu, Cedegao E. Zhang, Joshua B. Tenenbaum, Yoon Kim, and Roger P. Levy. 2025. On the same wavelength? evaluating pragmatic reasoning in language models across broad concepts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19913–19935, Suzhou, China. Association for Computational Linguistics. find
  26. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. find
  27. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  28. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktaschel, and Edward Grefenstette. 2022. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms. find
  29. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. Understanding figurative meaning through explainable visual entailment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1–23, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  30. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). find
  31. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  32. John R. Searle. 1975. Indirect Speech Acts, pages 59 – 82. Brill, Leiden, The Netherlands. find
  33. John R Searle. 1976. A classification of illocutionary acts. Language in society, 5(1):1–23. ↗
  34. Vidya Setlur and Melanie Tory. 2022. How do you converse with an analytical chatbot? revisiting gricean maxims for designing analytical conversational behavior. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. ↗
  35. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning (ICML). ↗ our card
  36. Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics (COLING). ↗
  37. Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. 2024. PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities. In Findings of the Association for Computational Linguistics: ACL 2024. find our card
  38. Settaluri Lakshmi Sravanthi, Kishan Maharaj, Sravani Gunnu, Abhijit Mishra, and Pushpak Bhattacharyya. 2025. Understand the implication: Learning to think for pragmatic understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23778–23790, Vienna, Austria. Association for Computational Linguistics. find
  39. Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd-Graber. 2024. Pregnant questions: The importance of pragmatic awareness in maternal health question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7253–7268, Mexico City, Mexico. Association for Computational Linguistics. ↗ our card
  40. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from language models. ↗ our card
  41. Jason Weston and Sainbayar Sukhbaatar. 2023. System 2 attention (is something you might need too). In Advances in Neural Information Processing Systems. ↗
  42. Ben Yao, Yazhou Zhang, Qiuchi Li, and Jing Qin. 2025. Is sarcasm detection a step-by-step reasoning process in large language models? Proceedings of the AAAI Conference on Artificial Intelligence. find our card
  43. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI models’ performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  44. Akhila Yerukola, Saujas Vaduguru, Daniel Fried, and Maarten Sap. 2024. Is the pope catholic? yes, the pope is catholic. generative evaluation of non-literal intent resolution in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 265–275, Bangkok, Thailand. Association for Computational Linguistics. ↗
  45. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650. find our card
  46. Kefan Yu, Qingcheng Zeng, Weihao Xuan, Wanxin Li, Jingyi Wu, and Rob Voigt. 2026. The pragmatic mind of machines: Tracing the emergence of pragmatic competence in large language models. ↗
  47. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  48. Yufei Tao, Ameeta Agrawal, Judit Dombi, Tetyana Sydorenko, and Jung In Lee. 2024. Chatgpt role-play dataset: Analysis of user motives and model naturalness. In International Conference on Language Resources and Evaluation. find
  49. Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor understanding challenge dataset for LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3517–3536, Bangkok, Thailand. Association for Computational Linguistics. find
  50. Shisen Yue, Siyuan Song, Xinyuan Cheng, and Hai Hu. 2024. Do large language models understand conversational implicature – a case study with a chinese sitcom. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics. find
  51. Zulipiye Yusupujiang and Jonathan Ginzburg. 2023. Unravelling indirect answers to wh-questions: Corpus construction, analysis, and generation. pages 336–348. find our card
  52. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. ↗
  53. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A grammar-based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074–2085, Online. Association for Computational Linguistics. ↗ our card
  54. Stephen Wan, Wei Liu, and Michael Strube. 2025. On the role of context for discourse relation classification in scientific writing. In Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025), pages 96–106, Suzhou, China. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/arai-2026-drinq-evaluating-conversational-implicature-controlled/