Elamur Research⌕
☰
home / papers / 2026 / Word Surprisal Correlates with Sentential Contradiction in LLMs

Word Surprisal Correlates with Sentential Contradiction in LLMs

Paper · 2026 · Ning Shi, Bradley Hauer, David Basil et al.

Summary

Shi et al. ask whether an LLM's word-level surprisal (the negative log probability of a word in context) tracks sentence-level contradiction between an NLI premise and hypothesis. Their hypothesis is that, when the hypothesis is conditioned on the premise, content words in contradicting hypotheses receive higher surprisal.

Modern models assign probabilities to subword tokens rather than words, so the authors build a token-to-word decoding algorithm that estimates word probabilities in open-vocabulary settings. They then test several design choices:

  • Context: the hypothesis alone, premise and hypothesis concatenated, or both placed in a causal template.
  • Surprisal: raw or relative to the most likely next word.
  • Aggregation: for example, last-word, maximum or mean surprisal over the hypothesis.

Across several datasets and open-source LLMs, surprisal and contradiction show a consistent, statistically significant positive correlation. A simple threshold classifier built on this signal beats chance. Causal framing makes the signal more reliable, even when the premise is factually false. The work links lexical uncertainty to sentential semantics and offers an interpretable view of LLM reasoning. It also exposes persistent model weaknesses, notably with negation.

Key points

  • Decoding algorithm: combines constrained beam search, dynamic normalization and simulated end-of-word (EOW) handling to turn token probabilities into word surprisal. Code and data are released on GitHub.
  • Context constructions: H-only, CAT and TEMP (a template of the form "Since {P}, therefore {H}"). Surprisal variants are Direct and Relative; aggregations include Last, Max and Mean.
  • Counterfactual test: Wiki-C* swaps premise and hypothesis in Wiki-C, so contradicting premises become false. Accuracy drops in every setting, but TEMP holds up best, while H-only and CAT suffer most from interference by the model's stored facts.
  • Imbalanced labels: on a resampled SNLI test set of 1,000 pairs (about 31% contradiction), Qwen with TEMP, Direct surprisal and Mean aggregation reaches roughly 71% accuracy. This edges past the ~69% majority baseline, with ROC-AUC around 72%.
  • Stated limitations: correlation is not causation, decoding is computationally costly, and the threshold is a tunable hyperparameter. Aggregation into one score also hides which specific words drive a contradiction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlbertaEdmonton · Canada44 works · 3,236 citations🔬Alberta Machine Intelligence InstituteAlberta · Canada4 works · 231 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Ning Shi, Bradley Hauer, David Basil, John Zhang and Grzegorz Kondrak. 2026. Word Surprisal Correlates with Sentential Contradiction in LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.211

@inproceedings{shi2026word,
  title = {{Word Surprisal Correlates with Sentential Contradiction in LLMs}},
  author = {Ning Shi, Bradley Hauer, David Basil, John Zhang and Grzegorz Kondrak},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.211},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.211}
}

References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bird. 2006. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72, Sydney, Australia. Association for Computational Linguistics.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
  3. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning, pages 38–45. find
  4. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches–erratum. Natural Language Engineering, 16(1):105–105. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine learning challenges workshop. find
  6. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio. Association for Computational Linguistics.
  7. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput…
  8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  9. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  10. Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast and contradiction in text processing. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1, AAAI’06, page 755–762. AAAI Press. find
  11. Bradley Hauer and Grzegorz Kondrak. 2022. WiC = TSV = WSD: On the equivalence of three semantic tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2478–2486, Seattle, United States. find
  12. Bradley Hauer and Grzegorz Kondrak. 2023. Taxonomy of problems in lexical semantics. In Findings of the Association for Computational Linguistics: ACL 2023. find
  13. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  14. Jerrold J. Katz. 1972. Semantic Theory. Harper & Row, New York. find
  15. Andrey Nikolaevich Kolmogorov. 1933. Foundations of probability theory. Julius Springer: Berlin, Germany. find
  16. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  17. Bill MacCartney. 2009. Natural language inference. Stanford University. find
  18. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140–156, Tilburg, The Netherlands. Association for Computational Linguistics. find
  19. Siddarth Mamidanna, Daking Rai, Ziyu Yao, and Yilun Zhou. 2025. All for one: Llms solve mental math at the last token with information transferred from other tokens. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  20. William Merrill, Alex Warstadt, and Tal Linzen. 2022. Entailment semantics can be extracted from an ideal language model. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 176–193, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational L… find
  21. William Merrill, Zhaofeng Wu, Norihito Naka, Yoon Kim, and Tal Linzen. 2024. Can you learn semantics through next-word prediction? the case of entailment. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2752–2773, Bangkok, Thailand. Association for Computational Linguis… find
  22. George A Miller. 1995. Wordnet: a lexical database for english. Comm. ACM, 38(11).
  23. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AA… find
  24. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa…
  25. Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, and Adhiguna Kuncoro. 2023. On “scientific debt” in NLP: A case for more rigour in language model pre-training research. In Proceedings of the 61st Annual Meeting of the Association for Computat… find
  26. Jingcheng Niu, Xingdi Yuan, Tong Wang, Hamidreza Saghir, and Amir H. Abdi. 2025. Llama see, llama do: A mechanistic perspective on contextual entrainment and distraction in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  27. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472, Miami, Florida, USA. Association for Comp… find
  28. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics.
  29. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  30. MohammadHossein Rezaei and Eduardo Blanco. 2024. Paraphrasing in affirmative terms improves negation understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 602–615, Bangkok, Thailand. Association for Computational Lin… find
  31. Alan Ritter, Stephen Soderland, Doug Downey, and Oren Etzioni. 2008. It‘s a contradiction – no, it‘s not: A case study using functional relations. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 11–20, Honolulu, Hawaii. Association for Computational L… find
  32. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  33. Ning Shi, Bradley Hauer, and Grzegorz Kondrak. 2024a. Lexical substitution as causal language modeling. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 120–132, Mexico City, Mexico. Association for Computational Linguistics. find
  34. Ning Shi, Bradley Hauer, Jai Riley, and Grzegorz Kondrak. 2024b. Paraphrase identification via textual inference. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 133–141, Mexico City, Mexico. Association for Computational Linguistics. find
  35. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  36. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: A…
  37. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Sl… find
  38. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  39. Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What’s the meaning of superhuman performance in today’s NLU? In Proceedings of the 61st Annual Meet… find
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  41. Zijun Wu, Zi Xuan Zhang, Atharva Naik, Zhijian Mei, Mauajama Firdaus, and Lili Mou. 2023. Weakly supervised explainable phrasal reasoning with neural fuzzy logic. In The Eleventh International Conference on Learning Representations. find
  42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  43. Johan van Benthem. 2008. A brief history of natural logic. In Mihir K. Chakraborty, Mohua Nath Mitra, Benedikt Löwe, and Sundar Sarukkai, editors, Logic, Navya-Nyāya & Applications: Homage to Bimal Krishna Matilal, pages 21–42. College Publications, London. Presented at the International Conference… find
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…
  45. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78–85.
  46. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks.
  47. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-word-surprisal-correlates-sentential-contradiction/