Elamur Research⌕
☰
home / papers / 2026 / Semantic Information: A Difference That Makes a Difference

Semantic Information: A Difference That Makes a Difference

Summary

The paper asks how semantic information in natural language can be measured empirically, without the truth-conditional, model-theoretic definitions common in philosophy. Building on Bateson's view of information as a meaningful difference, the authors define semantic information (SemI) as the drop in processing effort a language processor gains when it knows something about a document, such as its topic. Effort is modelled as surprisal. An uninformed processor relies on a generic word distribution, while informed processors use topic-specific distributions. SemI compares how far each processor's distribution diverges, via KL-divergence, from the document's actual word distribution, which captures the excess effort that remains. Semantic surprisal comes from the Topic Context Model (TCM), which builds on word–topic distributions. Tests on an English fairy tale corpus and two German news corpora supported the prediction: signs that carry prior information through semantic surprisal lower the processor's current surprisal. The authors conclude that semantic information grows with semantic prior information. The result is a corpus-driven, non-truth-functional measure that links distributional semantics, surprisal theory and Shannon information.

Key points

  • Core formula: SemI(τ_i) = log₂ KL(T, U) − log₂ KL(T, I_i).
    • T is the document's word distribution, U the uninformed distribution, and I_i the distribution informed by topic τ_i.
    • Variants are also discussed: one drops the logarithm, another uses cross-entropy instead of KL.
  • Effort and information: A document's average surprisal equals the cross-entropy H(T, P). Full information about a document corresponds to KL(T, T) = 0.
  • Separate concepts: Semantic surprisal, computed with TCM, is kept distinct from SemI and treated as its prerequisite.
  • Data:
    • English: the Fairy Tale Corpus (Vaz Lobo & de Matos, 2010).
    • German: two news corpora, with the language resources linked to Kölbl et al. (2020).
  • Missing details: The supplied text omits the experiments and results sections, so specific numbers and effect sizes could not be included.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4kfwe8i7ux33

@inproceedings{philipp2026semantic,
  title = {{Semantic Information: A Difference That Makes a Difference}},
  author = {Philipp, J. Nathanael and Kölbl, Max and Richter, Michael},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4kfwe8i7ux33},
  url = {https://doi.org/10.63317/4kfwe8i7ux33}
}

References

45 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gregory Bateson. 2000. Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology. University of Chicago press. find
  2. Martijn Bentum. 2021. Listening with great expectations: A study of predictive natural speech processing. Ph.D. thesis, [Sl]:[Sn]. find
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. find our card
  4. Alessandro Bondielli and Francesco Marcelloni. 2019. A survey on fake news and rumour detection techniques. Information sciences, 497:38– 55. find
  5. Marisa F. Boston, Shravan Vasishth, Richard L. Lewis, and Hiroko Drenhaus. 2008. Parsing costs as predictors of reading difficulty: An evaluation using the potsdam sentence corpus. Journal of Eye Movement Research, 2(1):1–12. find
  6. Harm Brouwer, Francesca Delogu, Noortje J Venhuizen, and Matthew W Crocker. 2021. Neurobehavioral correlates of surprisal in language comprehension: A neurocomputational model. Frontiers in Psychology, 12:615538. find
  7. Rudolf Carnap, Yehoshua Bar-Hillel, et al. 1952. An outline of a theory of semantic information. find
  8. David J Chalmers. 1997. The conscious mind: In search of a fundamental theory. Oxford Paperbacks. find
  9. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. In Proceedings of the 11th International Conference on Cognitive Modeling (ICCM), pages 213–218. Routledge. find
  10. Fred Dretske. 1981. Knowledge and the Flow of Information. MIT Press. find
  11. Anna Feldman and Jing Peng. 2013. Automatic detection of idiomatic clauses. In Computational Linguistics and Intelligent Text Processing: 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part I 14, pages 435–446. Springer. find
  12. JR Firth. 1957. A synopsis of linguistic theory 1930-1955. Studies in Linguistic Analysis, Special Volume/Blackwell. find
  13. Luciano Floridi. 2004. Outline of a theory of strongly semantic information. Minds and machines, 14:197–221. find
  14. Luciano Floridi. 2009. Philosophical conceptions of information. In Formal theories of information: From Shannon to semantic information theory and general concepts of information, pages 13– 53. Springer. find
  15. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1–8. Association for Computational Linguistics. find our card
  16. Zellig Harris. 1954. Distributional structure. Word, 10(2):146–162. find
  17. Bo Hu, Zhendong Mao, and Yongdong Zhang. 2025. An overview of fake news detection: From a new perspective. Fundamental research, 5(1):332–346. find
  18. T Florian Jaeger and Roger P Levy. 2007. Speakers optimize information density through syntactic reduction. In Advances in neural information processing systems, pages 849–856. find
  19. Artemy Kolchinsky and David H Wolpert. 2018. Semantic information, autonomous agency and non-equilibrium statistical physics. Interface focus, 8(6):20180041. find
  20. Max Kölbl, Yuki Kyogoku, J. Nathanael Philipp, Michael Richter, Clemens Rietdorf, and Tariq Keyword Extraction in Ger-Yousef. 2020. man: Information-theory vs. Deep Learning. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 1: NLPinAI, pages 459–464.… find
  21. Max Kölbl, Yuki Kyogoku, J. Nathanael Philipp, Michael Richter, Clemens Rietdorf, and Tariq Yousef. 2021. The semantic level of shannon information: Are highly informative words good keywords? a study on german. In Roussanka Loukanova, editor, Natural Language Processing in Artificial Intelligence -… find
  22. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  23. Roger Levy. 2011. Integrating surprisal and uncertain-input models in online sentence comprehension: formal techniques and empirical results. In Proceedings of the 49th annual meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1055– 1065. find
  24. Björn Lundgren. 2019. Does semantic information need to be truthful? Synthese, 196(7):2885– 2906. find
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26, pages 3111–3119. find
  26. Irene F. Monsalve, Roger Levy, and Edward Gibson. 2012. Frequency and surprisal in predictive human sentence processing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), pages 103– 111. Association for Computational Linguistics. find
  27. Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2018. Classifying idiomatic and literal expressions using topic models and intensity of emotions. arXiv preprint arXiv:1802.09961. — [arXiv:1802.09961](https://arxiv.org/abs/1802.09961) find
  28. J Nathanael Philipp, Max Kölbl, Erik Daas, Yuki Kyogoku, and Michael Richter. 2023a. Perplexed by idioms? In Knowledge Graphs: Semantics, Machine Learning, and Languages, pages 70–76. IOS Press. find
  29. J. Nathanael Philipp, Max Kölbl, Yuki Kyogoku, Tariq Yousef, and Michael Richter. 2022. One step beyond: Keyword extraction in german utilising surprisal from topic contexts. In Intelligent Computing, pages 774–786, Cham. Springer International Publishing. find
  30. J. Nathanael Philipp, Max Kölbl, and Michael Richter. 2025a. Surprisal in action: A comparative study of LDA and LSA for keyword extraction. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Long and Short Papers, pages 1–11, Hannover, Germany. HsH Applied Academic… find
  31. J. Nathanael Philipp, Olav Mueller-Reichau, Matthias Irmer, Michael Richter, and Max Kölbl. 2025b. Can information theory unravel the subtext in a chekhovian short story? In Proceedings of the 10th Workshop on Slavic Natural Language Processing (Slavic NLP 2025), pages 84–90, Vienna, Austria. Associ… find
  32. J. Nathanael Philipp, Michael Richter, Erik Daas, and Max Kölbl. 2023b. Are idioms surprising? In Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023), pages 149–154, Ingolstadt, Germany. Association for Computational Lingustics. find our card
  33. J. Nathanael Philipp, Michael Richter, Olav Mueller-Reichau, and Matthias Irmer. 2025c. Information theory unravels the subtext in chekhov. Digital Humanities Quarterly, 19(2). find
  34. J. Nathanael Philipp, Michael Richter, Tatjana Scheffler, and Roeland van Hout. 2024. The role of information in modeling german intensifiers. In Information structure and information theory, pages 117–145. Language Science Press. find
  35. Brian Roark, Asaf Bachrach, Carlos Cardenas, and Murat Kızılkaya. 2009. Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3… find
  36. Raphael Rubino, Ekaterina Lapshinova-Koltunski, and Josef van Genabith. 2016. Information density and quality estimation features as translationese indicators for human translation classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computationa… find
  37. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  38. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22–36. find
  39. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  40. Giulio Tononi. 2004. An information integration theory of consciousness. BMC Neuroscience, 5:42. find
  41. Peter D. Turney and Patrick Pantel. 2010. Distributional Semantics. The Handbook of Computational Linguistics and Natural Language Processing, Oxford. find
  42. Noortje J Venhuizen, Matthew W Crocker, and Harm Brouwer. 2019. Semantic entropy in language comprehension. Entropy, 21(12):1159. find
  43. Ethan G Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. 8. find our card
  44. Language Resource References Max Kölbl and Yuki Kyogoku and J. Nathanael Philipp and Michael Richter and Clemens Rietdorf and Tariq Yousef. 2020. Keyword Extraction in German: Information-theory vs. Deep Learning. INSTICC. SciTePress. PID https://doi.org/10.5220/0009374704590464. — [doi:10.5220/0009374704590464](https://doi.org/10.5220/0009374704590464) ↗
  45. Paula Vaz Lobo and David Martins de Matos. 2010. Fairy Tale Corpus Organization Using Latent Semantic Mapping and an Item-to-item Top-n Recommendation Algorithm. European Language Resources Association (ELRA). PID https://www.hlt.inesc-id.pt/w/Fairy_tale_corpus. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/philipp-2026-semantic-information-difference-that-makes/