Elamur Research⌕
☰
home / papers / 2026 / Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

Paper · 2026 · Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach et al.

Summary

The paper asks whether LLMs track beliefs that are communicated without being stated, and whether they revise those beliefs when a speaker later takes the implied meaning back. The authors model the common ground as a probability over propositions. An implicature makes a belief more likely than its negation, and a cancellation lowers that probability again. They build IMPLICATUREX, the first implicature-cancellation dataset: 271 items checked by experts and covering scalar, discourse and conversational implicatures, each paired with a cancelling continuation. Every item also has five crowdsourced human judgements. Model judgements are read off output probabilities. The strongest models reach human level on recognising scalar and discourse implicatures. They stay near chance on naturally-occurring conversational ones, and they fall far short of humans once recognition and cancellation are scored together. Control variants test three things: guessing the belief without seeing the utterance, cancellation by explicit negation, and follow-ups that strengthen the belief or leave it unchanged. These controls show that models lean on priors and keep beliefs more easily than they change them. The work exposes a concrete gap in how LLMs handle dynamic pragmatic reasoning.

Key points

  • Dataset sources. IMPLICATUREX draws on three sources:
    • 50 scalar items built on the ⟨some, all⟩ scale, drawn from Switchboard Corpus conversations as originally collected by Degen (2015), each given a hand-written cancellation;
    • 31 WSJ excerpts selected through implicit causal relations in the Penn Discourse Tree Bank (PDTB), recast as "discourse implicatures";
    • 197 conversational items, both synthetic two-turn and naturally-occurring multi-turn dialogue.
  • Cancellation vs. joint scoring. On naturally-occurring conversational items, Qwen 3 32B Thinking recognises cancellations at 0.84, against 0.92 for humans. When implicature and cancellation recognition are scored jointly, its accuracy is 0.76 on synthetic and 0.40 on natural items, compared with human scores of 0.90 and 0.72.
  • Keeping vs. changing beliefs. GPT-5.4 keeps a belief correctly 0.96 of the time after an irrelevant follow-up. It handles both strengthening and cancelling updates only 0.32 of the time.
  • Explicit negation. The IMPLICATURE⊥ variant rewrites cancellations as explicit negations. Accuracy rises but does not reach 1, except on scalar items, which suggests that negation handling itself causes some errors.
  • Prior-only control. Models often score the implied belief highly without seeing the context or the utterance, especially for ⟨some, all⟩ items. This casts doubt on how much of their success reflects genuine pragmatic inference.

Interesting ideas

  • I1. Defining an implicature as a shift in belief probability, and a cancellation as a measurable drop in that probability, turns a vague pragmatic notion into a testable before/after comparison.
  • I2. Pairing each stimulus with three kinds of follow-up (cancelling, neutral, strengthening) separates "never moves" from "moves correctly", and it reveals a bias toward inertia that a single test type would hide.
  • I3. A prior-only control removes the triggering utterance and checks whether the system still asserts the belief, which exposes cases where correct answers come from world priors rather than the text.
  • I4. Reframing implicit causal discourse relations from PDTB as implicatures, then appending a "though…"-style cancelling sentence, yields naturally-occurring pragmatic test items cheaply from existing annotation.
  • I5. Writing an explicit-negation version of each cancellation separates pragmatic failure from plain negation-processing failure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Each implied proposition in the text world could carry a graded strength that later utterances raise or lower, so the explanation trace shows exactly which sentence weakened a belief.
  • I2 (answer): Fable and tale QA tests could include neutral, reinforcing and retracting continuations, to check that the reader updates its answers only when it should.
  • I3 (learn): This fits the negative-control evaluation directly: rerun implicature and QA items with the key sentence ablated and flag any "success" that survives.
  • I4 (frames): The prag crate could harvest implicit causal links from discourse-annotated text as gold implicature cases, with cancellation markers like "though" triggering a defeat rule.
  • I5 (frames): Every implicature-cancellation rule in prag could be paired with an explicit "not" variant, so errors can be attributed to the negation handling in the UD-to-frame mapping rather than to the inference rule.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksImplicature · 138 works

Cite

Cite as Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach and Jackie Chi Kit Cheung. 2026. Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation. arXiv:2607.25094.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.25094 · arXiv 2607.25094

@misc{piano2026evaluating,
  title = {{Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation}},
  author = {Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach and Jackie Chi Kit Cheung},
  year = {2026},
  doi = {10.48550/arxiv.2607.25094},
  eprint = {2607.25094},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.25094}
}

References

50 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F Allen and C Raymond Perrault. 1980. Analyzing intention in utterances. Artificial intelligence, 15(3):143–178. ↗
  2. Catherine Anderson. 2018. Essentials of linguistics. McMaster University. find
  3. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1173– 1182.
  4. Luciana Benotti. 2010. Implicature as an Interactive Process. Ph.D. thesis, Université Henri Poincaré-Nancy I. ↗
  5. Richard Breheny, Napoleon Katsos, and John Williams. 2006. Are generalised scalar implicatures generated by default? An on-line investigation into the role of context in generating pragmatic inferences. Cognition, 100(3):434–463. find
  6. Ted Byrt, Janet Bishop, and John B Carlin. 1993. Bias, prevalence and kappa. Journal of clinical epidemiology, 46(5):423–429. ↗
  7. Ye-eun Cho and Seong mook Kim. 2024. Pragmatic inference of scalar implicature by LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 10–20.
  8. Herbert H Clark and Susan E Brennan. 1991. Grounding in communication. Perspectives on Socially Shared Cognition. ↗
  9. Herbert H Clark and Meredyth A Krych. 2004. Speaking while monitoring addressees for understanding. Journal of memory and language, 50(1):62–81. ↗
  10. Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. ↗
  11. Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin, 114(3):494. ↗
  12. Yan Cong. 2024. Manner implicatures in large language models. Scientific Reports, 14(1):29113. ↗
  13. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive science, 19(2):233–263. ↗
  14. Kees van Deemter, Albert Gatt, Ielka van der Sluis, and Richard Power. 2012. Generation of referring expressions: Assessing the incremental algorithm. Cognitive science, 36(5):799–836. ↗
  15. Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. Semantics and Pragmatics, 8:11–1. ↗
  16. Anita Fetzer. 2018. The linguistic realization of contrastive discourse relations in context: Contextualization and discourse common ground. Modélisation et utilisation du contexte. ↗
  17. Gemma-Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  18. Elizabeth Jasmi George and Radhika Mamidi. 2020. Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316–2323. ↗
  19. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ICASSP’92, page 517–520, USA. IEEE Computer Society. find
  20. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  21. H Paul Grice. 1957. Meaning. The philosophical review, 66(3):377–388. find
  22. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  23. Daniel J Grodner, Natalie M Klein, Kathleen M Carbary, and Michael K Tanenhaus. 2010. “Some,” and possibly all, scalar inferences are not delayed: Evidence for immediate pragmatic enrichment. Cognition, 116(1):42–55. ↗
  24. Barbara J Grosz and Candace L Sidner. 1986. Attention, intentions, and the structure of discourse. Computational linguistics, 12(3):175–204. ↗
  25. Irene Roswitha Heim. 1982. The semantics of definite and indefinite noun phrases. University of Massachusetts Amherst. ↗
  26. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4194–4213.
  27. Yi Ting Huang and Jesse Snedeker. 2018. Some inferences still take time: Prosody, predictability, and the speed of scalar implicatures. Cognitive psychology, 102:105–126. ↗
  28. Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (Comet-)Atomic 2020: on symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, pages 6384–6392. ↗
  29. Ellen A Isaacs and Herbert H Clark. 1987. References in conversation between experts and novices. Journal of experimental psychology: general, 116(1):26. find
  30. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705.
  31. Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In Proceedings of the 29th International Conference on Computational Linguistics, pages 779–785, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  32. Hans Kamp. 1981. A theory of truth and semantic representation. Proceedings of the Third Amsterdam Colloquium, pages 277–322. find
  33. Anita Körner, Antonia Tolzin, Andreas Janson, Jan Marco Leimeister, and Ralf Rummer. 2025. Common ground improves learning with conversational agents. Behaviour & Information Technology, pages 1–17. ↗
  34. Alex Lascarides and Nicholas Asher. 1991. Discourse relations and defeasible knowledge. In 29th Annual Meeting of the Association for Computational Linguistics, pages 55–62.
  35. David Lewis. 1979. Scorekeeping in a language game. Journal of philosophical logic, 8(1):339–359. ↗
  36. Annie Louis, Dan Roth, and Filip Radlinski. 2020. “I’d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425.
  37. Ira A Noveck. 2001. When children are more logical than adults: Experimental investigations of scalar implicature. Cognition, 78(2):165–188. ↗
  38. Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. Web Download. LDC2019T05. ↗
  39. Qwen-Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. ↗
  40. Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675. find
  41. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by LLMs. Advances in Neural Information Processing Systems, 36:20827–20905.
  42. Reinhard Selten and Massimo Warglien. 2007. The emergence of simple languages in an experimental coordination game. Proceedings of the National Academy of Sciences, 104(18):7361–7366. ↗
  43. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 292–314, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. ↗
  44. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and Cognition. find
  45. Robert Stalnaker. 1998. On the representation of context. Journal of logic, language and information, 7(1):3–19. ↗
  46. Elizabeth S Veinott, Judith Olson, Gary M Olson, and Xiaolan Fu. 1999. Video helps remote work: Speakers who need to negotiate common ground benefit from seeing each other. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 302–309. find
  47. Alexander C Wilson and Dorothy VM Bishop. 2021. “Second guessing yourself all the time about what they really mean...”: Cognitive differences between autistic and non-autistic adults in understanding implied meaning. Autism Research, 14(1):93–101. find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  49. Xiao Yang, Utako Minai, and Robert Fiorentino. 2018. Context-sensitivity and individual differences in the derivation of scalar implicature. Frontiers in psychology, 9:1720. ↗
  50. Shisen Yue, Siyuan Song, Xinyuan Cheng, and Hai Hu. 2024. Do large language models understand conversational implicature - A case study with a chinese sitcom. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), pages 1270–1285.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/piano-2026-evaluating-communicative-belief-updates-large/