Elamur Research⌕
☰
home / papers / 2026 / CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models

CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models

Paper · 2026 · Jon Chun, Hannah Sussman, Adrian Mangine et al.

Summary

Language models often miss what a speaker actually means when the wording points one way and the intent another. The CEI (Contextual Emotional Inference) Benchmark tests this. It contains 300 researcher-written scenarios. Each one gives a situation, a speaker and a listener with a stated power relation (peer, higher→lower, lower→higher), and an ambiguous utterance. The model must infer the speaker's emotion. The scenarios are set in workplace, family, social and service contexts and are divided into five categories of pragmatic phenomenon: passive aggression, deflection/misdirection, sarcasm/irony, strategic politeness and mixed signals.

Three trained annotators labeled every item with Plutchik emotion categories and VAD ratings. Their agreement is low (Fleiss' κ 0.06–0.25), and the authors treat that as a property of the task rather than as noise. A four-level quality-control pipeline separates real errors from legitimate disagreement.

Seven LLMs were tested. The best reaches about 25% accuracy, against 54% human majority agreement, and neither chain-of-thought nor few-shot prompting closes the gap. This shows that socially situated pragmatic inference remains an open weakness of current models.

Key points

  • Dataset size and coverage: 300 scenarios, 5 pragmatic subtypes, 3 power configurations and 4 social settings. Each scenario has 3 independent annotations, 900 in total.
  • Labels and agreement: each scenario has categorical labels (Plutchik taxonomy) and dimensional labels (valence, arousal, dominance).
    • Categorical agreement is measured with Fleiss' κ, which ranges from 0.06 to 0.25 by subtype.
    • VAD agreement is measured with ICC.
    • Majority-vote gold labels are provided, and every individual annotator label is also released.
  • Quality control: the pipeline runs automated schema validation, statistical outlier detection and agreement analysis. Flagged items then go to expert adjudication by a meta-annotator.
  • Baselines: 7 LLMs (4 commercial, 3 open-weight) were run zero-shot, with CoT and few-shot.
    • The best model scores about 25%, versus 54% human majority agreement.
    • Human and model difficulty profiles diverge. Sarcasm is easiest for people but hardest for models, and deflection shows the opposite pattern.
  • Release and reproducibility:
    • Data is CC-BY-4.0 and code is MIT, at GitHub (jon-chun/cei-tom-dataset-base), HuggingFace (jonc/cei-benchmark) and Zenodo.
    • The dataset is versioned as v1.0.0.
    • The local analysis pipeline reproduces in roughly 30 seconds without API calls, using seed 42.

Interesting ideas

  • I1. Power direction between speaker and listener is an explicit input that shifts how an utterance should be read, not something to predict; this follows Brown and Levinson's idea that indirectness grows with power distance and face threat.
  • I2. The authors keep every annotator's label instead of collapsing to one gold answer, because they treat disagreement as a reproducible signal of genuinely debatable items.
  • I3. Test items are chosen on purpose so that surface cues mislead, which makes the benchmark diagnose real inference rather than keyword matching.
  • I4. Quality control is staged: cheap automatic checks (schema, outliers, agreement) come first, and only flagged items go to a costly expert.
  • I5. Per-subtype difficulty profiles of humans and models are compared to show that model errors are qualitatively different, not just a copy of human uncertainty.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could attach a speaker→listener power feature to each speech-act frame and let implicature and irony rules depend on it, so "Sure, I'll do it" from a subordinate can be read as reluctant compliance.
  • I2 (learn): when witness parsers disagree on silver data, Mova could store the full vote distribution and use it as a soft target or a "debatable" flag, rather than discarding the item or forcing a majority.
  • I3 (answer): fable and school-test QA sets could include controls where the literal reading gives the wrong answer, so readers get credit only for actually using context.
  • I4 (learn): the same funnel could decide which silver trees or frames go up to a higher LLM teacher tier, saving teacher calls for the cases automatic checks cannot settle.
  • I5 (learn): evaluation reports could break pragmatic accuracy down by phenomenon and compare it against human-agreement levels, revealing where Mova's rules fail systematically and where items are simply ambiguous.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Kenyon College.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Jon Chun, Hannah Sussman, Adrian Mangine, Murathan Kocaman, Kirill Sidorko, Abhigya Koirala, Andre McCloud, Gwen Eisenbeis, Wisdom Akanwe, Moustapha Gassama, Eliezer Gonzalez Chirinos, Anne-Duncan Enright, Peter Dunson, Tiffanie Ng, Anna von Rosenstiel and Godwin Idowu. 2026. CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models. arXiv:2603.09993.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.09993 · arXiv 2603.09993

@misc{chun2026cei,
  title = {{CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models}},
  author = {Jon Chun, Hannah Sussman, Adrian Mangine, Murathan Kocaman, Kirill Sidorko, Abhigya Koirala, Andre McCloud, Gwen Eisenbeis, Wisdom Akanwe, Moustapha Gassama, Eliezer Gonzalez Chirinos, Anne-Duncan Enright, Peter Dunson, Tiffanie Ng, Anna {von Rosenstiel} and Godwin Idowu},
  year = {2026},
  doi = {10.48550/arxiv.2603.09993},
  eprint = {2603.09993},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.09993}
}

References

70 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of EMNLP, pages 1173–1182, 2016. doi: 10.18653/v1/D16-1125. ↗
  2. Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596, 2008. doi: 10.1162/coli.07-034-R2. ↗
  3. Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018. doi: 10.1162/tacl a 00041. ↗
  4. Laura-Ana-Maria Bostan and Roman Klinger. An analysis of annotated corpora for emotion classification in text. In Proceedings of COLING, pages 2104–2119, 2018. find
  5. Margaret M. Bradley and Peter J. Lang. Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, Center for Research in Psychophysiology, University of Florida, 1999. find
  6. Philip Bramsen, Martha Escobar-Molano, Ami Patel, and Rafael Alonso. Extracting social power relationships from natural language. In Proceedings of ACL, pages 773–782, 2011. find
  7. Penelope Brown and Stephen C. Levinson. Politeness: Some Universals in Language Usage. Cambridge University Press, 1987.
  8. Sven Buechel and Udo Hahn. Emotion analysis as a regression problem – dimensional models and their implications on emotion representation and metrical evaluation. In Proceedings of ECAI, pages 1114–1122, 2016. ↗
  9. Sven Buechel and Udo Hahn. EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of EACL, pages 578–585, 2017. doi: 10.18653/v1/E17-2092. ↗
  10. Sven Buechel, Luise Modersohn, and Udo Hahn. Towards label-agnostic emotion embeddings. In Proceedings of EMNLP, pages 9231–9249, 2021. doi: 10.18653/v1/2021. emnlp-main.728. ↗
  11. Carlos Busso et al. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359, 2008. doi: 10.1007/s10579-008-9076-6. ↗
  12. Jon Chun, Hannah Sussman, Adrian Mangine, Murathan Kocaman, Kirill Sidorko, Abhigya Koirala, Andre McCloud, Gwen Eisenbeis, Wisdom Akanwe, Moustapha Gassama, Eliezer Gonzalez Chirinos, Anne-Duncan Enright, Peter Dunson, Tiffanie Ng, Anna von Rosenstiel, and Godwin Idowu. CEI benchmark: Pragmatic reasoning dataset, 2026. URL https://doi.org/10.5281/zenodo.18528706. ↗
  13. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. A computational approach to politeness with application to social factors. In Proceedings of ACL, pages 250–259, 2013. doi: 10.3115/v1/P13-1025. ↗
  14. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of ACL, pages 4040–4054, 2020. doi: 10.18653/v1/2020.acl-main.372. ↗
  15. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382, 1971. doi: 10.1037/h0031619. ↗
  16. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Understanding social reasoning in language models with language models. In Proceedings of NeurIPS, 2023. ↗
  17. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, et al. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. doi: 10.1145/3458723. ↗
  18. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 1–11, 2020. doi: 10.18653/v1/2020.figlang-1.1. ↗
  19. Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818–829, 2016. doi: 10.1016/j.tics. 2016.08.005. ↗
  20. H. Paul Grice. Logic and conversation. Syntax and Semantics, 3:41–58, 1975. ↗
  21. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of ACL, pages 4194–4213, 2023. doi: 10.18653/v1/2023.acl-long. 230. ↗
  22. Junhyung Hyun and Youngjoong Kim. DiPlomat: A dialogue dataset for situated pragmatic reasoning. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2023.
  23. Paloma Jeretič, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of ACL, pages 8690–8705, 2020. doi: 10.18653/v1/2020.acl-main.768. ↗
  24. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):1–22, 2017. doi: 10.1145/3124420. ↗
  25. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. A large self-annotated corpus for sarcasm. In Proceedings of LREC, 2018.
  26. Hyunwoo Kim et al. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of EMNLP, 2023. doi: 10.18653/v1/2023.emnlp-main.890. ↗
  27. Michal Kosinski. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024. doi: 10.1073/pnas. 2405460121. ↗
  28. Klaus Krippendorff. Content Analysis: An Introduction to Its Methodology. SAGE Publications, 4th edition, 2018. ↗
  29. Label Studio. Label studio: Data labeling software, 2020. URL https://labelstud.io. ↗
  30. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Proceedings of EMNLP-IJCNLP, pages 5872–5877, 2019. doi: 10.18653/v1/D19-1598. ↗
  31. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Proceedings of KR, 2012. find
  32. Stephen C. Levinson. Pragmatics. Cambridge University Press, 1983. find
  33. Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, Jiangyan Yi, Rui Liu, Kele Xu, Bin Liu, Erik Cambria, Guoying Zhao, Björn W. Schuller, and Jianhua Tao. MER 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition. In Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing (MRAC ’24), 2024. doi: 10.1145/3689092.3689959. ↗
  34. Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, and Jianhua Tao. OV-MER: Towards open-vocabulary multimodal emotion recognition. In Proceedings of ICML, 2025. ↗
  35. Percy Liang et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. find
  36. Dongjun Lim and Yun-Gyung Cheong. Integrating Plutchik’s theory with mixture of experts for enhancing emotion classification. In Proceedings of EMNLP, 2024. doi: 10.18653/v1/ 2024.emnlp-main.50. ↗
  37. Yuxuan Liu, Yang Zhao, Ximing Li, and Qin Zhang. EmoBench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. find
  38. Albert Mehrabian. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology, 14(4):261–292, 1996. doi: 10.1007/BF02686918. ↗
  39. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. SemEval-2018 task 1: Affect in tweets. In Proceedings of SemEval, pages 1–17, 2018. doi: 10.18653/v1/S18-1001. ↗
  40. Saif M. Mohammad. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of ACL, pages 174–184, 2018. doi: 10.18653/ v1/P18-1017. ↗
  41. Saif M. Mohammad and Peter D. Turney. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465, 2013. doi: 10.1111/j.1467-8640.2012. 00460.x. ↗
  42. Minxue Niu, Yara El-Tawil, Amrit Romana, and Emily Mower Provost. Rethinking emotion annotations in the era of large language models. IEEE Transactions on Affective Computing, 2024. doi: 10.1109/TAFFC.2024.3519178. ↗
  43. Silviu Oprea and Walid Magdy. iSarcasm: A dataset of intended sarcasm. In Proceedings of ACL, pages 1279–1289, 2020. doi: 10.18653/v1/2020.acl-main.118. ↗
  44. Rebecca J. Passonneau and Bob Carpenter. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326, 2014. doi: 10.1162/tacl a 00185. ↗
  45. Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694, 2019. doi: 10.1162/tacl a 00293. ↗
  46. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research, 22(164):1–20, 2021. find
  47. Steven Pinker, Martin A. Nowak, and James J. Lee. The logic of indirect speech. Proceedings of the National Academy of Sciences, 105(3):833–838, 2008. doi: 10.1073/pnas. 0707192105. ↗
  48. Barbara Plank, Dirk Hovy, and Anders Søgaard. Linguistically debatable or just plain wrong? In Proceedings of ACL, pages 507–511, 2014. doi: 10.3115/v1/P14-2083. ↗
  49. Robert Plutchik. A general psychoevolutionary theory of emotion. Emotion: Theory, Research, and Experience, 1:3–33, 1980. ↗
  50. Robert Plutchik. The nature of emotions. American Scientist, 89(4):344–350, 2001. ↗
  51. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, et al. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of ACL, pages 527–536, 2019. doi: 10.18653/v1/P19-1050. ↗
  52. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. Predicting overt display of power in written dialogs. In Proceedings of NAACL-HLT, pages 518–522, 2012. find
  53. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. Event2mind: Commonsense inference on events, intents, and reactions. In Proceedings of ACL, pages 463–473, 2018. doi: 10.18653/v1/P18-1043. ↗
  54. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011. find
  55. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by LLMs. In Proceedings of NeurIPS, 2023.
  56. James A. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980. doi: 10.1037/h0077714. ↗
  57. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of EMNLP-IJCNLP, pages 4463–4473, 2019. doi: 10.18653/v1/D19-1454. ↗
  58. Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? On the limits of social intelligence in large LMs. In Proceedings of EMNLP, pages 3762–3780, 2022. doi: 10.18653/v1/2022.emnlp-main.248. ↗
  59. John R. Searle. Indirect speech acts. In Syntax and Semantics, volume 3, pages 59–82. Academic Press, 1975. ↗
  60. Natalie Shapira et al. Clever Hans or neural theory of mind? Stress testing social reasoning in large language models. In Proceedings of EACL, 2024. find
  61. Patrick E. Shrout and Joseph L. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420–428, 1979. doi: 10.1037/0033-2909.86.2.420. ↗
  62. Dan Sperber and Deirdre Wilson. Relevance: Communication and Cognition. Harvard University Press, Cambridge, MA, 1986. find
  63. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
  64. Carlo Strapparava and Rada Mihalcea. SemEval-2007 task 14: Affective text. In Proceedings of SemEval, pages 70–74, 2007.
  65. Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. ↗
  66. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470, 2021. doi: 10.1613/jair.1.12752. ↗
  67. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4):1191–1207, 2013. doi: 10.3758/s13428-012-0314-x. ↗
  68. Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Findings of EMNLP, pages 10691–10706, 2023. doi: 10.18653/v1/2023. findings-emnlp.717. ↗
  69. Hainiu Xu et al. OpenToM: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. In Proceedings of ACL, 2024. find
  70. Amir Zadeh et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of ACL, pages 2236–2246, 2018. doi: 10.18653/v1/P18-1208. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chun-2026-cei-benchmark-evaluating-pragmatic-reasoning/