Elamur Research⌕
☰
home / papers / 2026 / What does Surprisal have to do with Information Status?

What does Surprisal have to do with Information Status?

Paper · 2026 · Andrew Thomas Dyer

Summary

The paper asks whether language-model surprisal can stand in for information status, meaning how given or new a mention is in discourse. Many researchers treat low surprisal as a sign of givenness. The author tests this assumption on CiepInf, a parallel literary corpus in nine languages. It was parsed into UD with UDPipe 2 and has mentions annotated for information status in CorefUD format.

Two regression studies were run with scikit-learn:

  • Experiment 1: a logistic regression predicts information status from UPOS and deprel.
  • Experiment 2: a Poisson regression predicts the surprisal of each mention's head word from information status, UPOS and word frequency. Surprisal comes from the multilingual autoregressive model mGPT.

The results:

  • Part of speech alone recovers information status very well.
  • Information status by itself barely explains surprisal.
  • UPOS explains surprisal better than information status in most languages, and word frequency explains it best.

The conclusion is that surprisal should not be read as a continuous givenness score. Any claimed link between the two must first be separated from cheap, context-free cues.

Key points

  • Data: The nine languages in CiepInf are Ukrainian, Turkish, Hindi, Greek, Indonesian, Portuguese, German, English and Chinese. Ukrainian has about 900 sentences, with roughly 1310 new and 1042 old mentions. Mentions were extracted with Udapi.
  • Experiment 1 (UPOS → information status): F1 ranges from .75 (Portuguese) to .98 (German), with .83 for Ukrainian. Deprel is weaker, and adding it or its interaction with UPOS changes almost nothing.
  • Experiment 2 (predicting surprisal):
    • Information status alone gives R2 between .02 and .16.
    • UPOS alone reaches up to .21.
    • Frequency from wordfreq (zipf_frequency) reaches .22–.42.
    • Adding the infstat×UPOS interaction to frequency gives no gain.
    • Chinese was left out of the frequency models because the wordfreq lookup did not work for it.
  • Direction of effects: pronouns lean given and nouns lean new across all languages. Proper nouns go both ways, though in Greek they lean noticeably toward new.
  • Setup:
    • mGPT was run with a 256-token context and a stride of 128.
    • Eight-fold cross-validation was used with the default L2 penalty.
    • The study is limited to one model and one hyperparameter setting.

Interesting ideas

  • I1. Before crediting a contextual signal with discourse knowledge, the author checks how much of it a trivial POS-only model already explains; this "mundane-cue baseline" guards against mistaking word-class effects for deep understanding.
  • I2. Subword tokens were forced to split only inside UD token boundaries, then merged back with their surprisals summed, so model scores line up exactly with treebank words.
  • I3. Mentions of varying length are scored through their syntactic head rather than the full span, which gives one comparable unit per mention.
  • I4. Pronoun-versus-noun form is a strong and nearly language-independent cue for given versus new referents, so a POS rule makes a solid, explainable first guess.
  • I5. Using Poisson rather than linear regression suits a target that is non-negative and long-tailed, such as surprisal.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every evaluation of a discourse-sensitive component should include a negative-control run that uses only UPOS/deprel features, so any claimed gain is measured beyond that floor.
  • I2 (tree): gparse or any transformer feature extractor can use the same rule so that per-word scores attach cleanly to UD nodes without alignment heuristics.
  • I3 (frames): Coreference and givenness features in prag can be keyed on the mention head from the UD tree, which keeps span-length noise out of the decisions.
  • I4 (world): When adding an entity to the world of a text, a pronoun can default to linking with an existing entity and a full noun to creating a new one, with context then confirming or overriding the guess.
  • I5 (learn): Diagnostic analyses of skewed quantities like parser error counts or confidence scores can use a Poisson model, which fits them more faithfully.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 102 works

Cite

Cite as Andrew Thomas Dyer. 2026. What does Surprisal have to do with Information Status?. Proceedings of the 8th Workshop on Research in Computational Linguistic Typology and Multilingual NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigtyp-main.4

@inproceedings{dyer2026what,
  title = {{What does Surprisal have to do with Information Status?}},
  author = {Andrew Thomas Dyer},
  booktitle = {Proceedings of the 8th Workshop on Research in Computational Linguistic Typology and Multilingual NLP},
  year = {2026},
  doi = {10.18653/v1/2026.sigtyp-main.4},
  url = {https://doi.org/10.18653/v1/2026.sigtyp-main.4}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. 2013. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pages 108–122. ↗
  2. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. ↗
  3. Wallace L. Chafe. 1976. Givenness, contrastiveness, definiteness, subjects, topics, and point of view. In Charles N. Li, editor, Subject and Topic, pages 25–55. Academic Press, New York. find
  4. Maciej Ogrodniczuk, Michal Novak, Massimo Poesio, Sameer Pradhan, and Vincent Ng, editors. 2025. Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference. Association for Computational Linguistics, Suzhou, China. find
  5. Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2644–2663, St. Julian’s, Malta. Association for Computational Linguistics. find
  6. Martin Popel, Zdeněk Žabokrtský, and Martin Vojtek. 2017. Udapi: Universal API for Universal Dependencies. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 96–101, Gothenburg, Sweden. Association for Computational Linguistics.
  7. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-shot learners go multilingual. Transactions of the Association for Computational Linguistics, 12:58–79. ↗
  8. Robyn Speer. 2022. wordfreq: v3.0. find
  9. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207, Brussels, Belgium. Association for Computational Linguistics.
  10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
  11. Annemarie Verkerk and Luigi Talamo. 2024. mini-CIEP+ : A shareable parallel corpus of prose. In Proceedings of the 17th Workshop on Building and Using Comparable Corpora (BUCC) @ LREC-COLING 2024, pages 135–143, Torino, Italia. ELRA and ICCL. find
  12. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. ↗
  13. Weijie Xu and Richard Futrell. 2024. Syntactic dependency length shaped by strategic memory allocation. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 1–9, St. Julian’s, Malta. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dyer-2026-what-does-surprisal-have-do/