Elamur Research⌕
☰
home / papers / 2026 / Modeling the Memory-Surprisal Trade-Off over Time: Communicative Efficiency Decreases with Lexico-Grammatical Change in Scientific English

Modeling the Memory-Surprisal Trade-Off over Time: Communicative Efficiency Decreases with Lexico-Grammatical Change in Scientific English

Summary

The paper asks whether communicative efficiency, measured with the memory-surprisal trade-off (MST), stays stable as a language variety changes. Its case study is scientific English, which over three centuries moved from clause-heavy sentences to compact, information-dense noun phrases while its vocabulary grew sharply. The authors compare the Royal Society Corpus (RSC) with the Corpus of Historical American English (COHA), which serves as a general-language baseline. For each year, they train language models on subsampled data of roughly equal size and derive MST curves. Efficiency is scored as the area under the curve (AUC), computed per decade, per 50-year period, and separately for nouns, verbs and other words. They also examine the slopes of the curves. Linear mixed-effects models test how time, register and word class affect the results. Over time, the growing vocabulary pushes minimal surprisal upward, so efficiency declines. Shorter, more local nominal dependencies also change where predictive information sits across memory budgets. Both effects are stronger in RSC than in COHA. This appears to be the first use of the MST to study diachronic and register variation within a single language.

Key points

  • The data are RSC, covering scientific writing from the 18th to the 20th century, and COHA, with both corpora analysed between 1750 and 2000. The training subsamples contain about 2.5M tokens for RSC and 3.5M for COHA, with 10% held out for development.
  • Surprisal was estimated with three tokenization schemes: word-level, lempos (lemma combined with UPOS) and BPE. Vocabularies were about 74K–98K items for the word-level and lempos schemes and 100K for BPE. Rare word-level items were replaced by POS-specific unknown tokens.
  • The illustrative RSC curves show the lowest AUC in 1820 and the highest in 1990. The 1950 curve crosses the 1850 curve.
  • The number of noun types in RSC grows roughly exponentially from around 1900, which drives the rise in minimal surprisal.
  • LME effect sizes differ widely across the three tokenizations, but the direction of every effect is the same.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4txotk7fwkhp

@inproceedings{steuer2026modeling,
  title = {{Modeling the Memory-Surprisal Trade-Off over Time: Communicative Efficiency Decreases with Lexico-Grammatical Change in Scientific English}},
  author = {Steuer, Julius and Krielke, Marie-Pauline and Degaetano-Ortlieb, Stefania and Teich, Elke and Klakow, Dietrich},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4txotk7fwkhp},
  url = {https://doi.org/10.63317/4txotk7fwkhp}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Banks. 2003. The evolution of grammatical metaphor in scientific writing. Amsterdam Studies in the Theory and History of Linguistic Science Series 4, pages 127–148. ↗
  2. Douglas Biber and Victoria Clark. 2002. Historical shifts in modification patterns with complex noun phrase structures. English Historical Morphology. Selected Papers from 11 ICEHL, Santiago de Compostela, 7–11 September 2000, 11:43–66. ↗
  3. Douglas Biber and Bethany Gray. 2011. Grammatical change in the noun phrase: The influence of written language use. English Language and Linguistics, 15(2):223–250. ↗
  4. TM Cover and Joy A Thomas. 2006. Elements of information theory. Hoboken, NJ: Wiley-Interscience. ↗
  5. Mark Davies. 2021. Corpus of Historical American English (COHA). find
  6. Stefania Degaetano-Ortlieb, Hannah Kermes, Ashraf Khamis, and Elke Teich. 2019. An Information-Theoretic Approach to Modeling Diachronic Change in Scientific English. In From Data to Evidence in English Language Research, Language and Computers, pages 258–281. Brill. ↗
  7. Stefan Fischer, Jörg Knappen, Katrin Menzel, and Elke Teich. 2020. The Royal Society Corpus 6.0: Providing 300+ Years of Scientific Writing for Humanistic Study. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 794–802, Marseille, France. European Language Resources Association. find
  8. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336–10341. National Acad Sciences. ↗
  9. Edward Gibson. 2000. The dependency locality theory: a distance-based theory of linguistic complexity. In Image, language, brain: Papers from the first Mind Articulation Project Symposium, pages 95–126. Cambridge, MA: MIT Press. ↗
  10. Kristina Gulordava and Paola Merlo. 2015. Diachronic Trends in Word Order Freedom and Dependency Length in Dependency-Annotated Corpora of Latin and Ancient Greek. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 121–130. Uppsala University, Uppsala, Sweden. find
  11. Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal. Psychological Review, 128(4):726– 756. ↗
  12. Michael Hahn and Richard Futrell. 2020. Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory and Surprisal. Society for Computation in Linguistics, 3(1). ↗
  13. M.A.K. Halliday and J.R. Martin. 1993. Writing science: Literacy and discursive power. Falmer Press. find
  14. Michael A. K. Halliday. 1988. On the language of physical science. In Mohsen Ghadessy, editor, Registers of written English: Situational factors and linguistic features, pages 162–177. Pinter. find
  15. Marianne Hundt, David Denison, and Gerold Schneider. 2012. Relative complexity in scientific discourse. English Language and Linguistics, 16(2):209–240. ↗
  16. Tom S Juzek, Marie-Pauline Krielke, and Elke Teich. 2020. Exploring diachronic syntactic shifts with dependency length: the case of scientific English. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 109–119. Association for Computational Linguistics. find our card
  17. Marie-Pauline Krielke. 2024. Cross-linguistic Dependency Length Minimization in scientific language: Syntactic complexity reduction in English and German in the Late Modern period. Languages in Contrast, 24(1):133–163. ↗
  18. Marie-Pauline Krielke, Luigi Talamo, Mahmoud Fawzi, and Jörg Knappen. 2022. Tracing Syntactic Change in the Scientific Genre: Two Universal Dependency-parsed Diachronic Corpora of Scientific English and German. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4808–4816, Marseille, France. European Language Resources Association. ↗ our card
  19. Lei Lei and Ju Wen. 2020. Is dependency distance experiencing a process of minimization? A diachronic study based on the State of the Union addresses. Lingua, 239:102762. find
  20. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. ↗
  21. Richard L. Lewis and Shravan Vasishth. 2005. An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval. Cognitive Science, 29(3):375–419. ↗
  22. Haitao Liu. 2008. Dependency Distance as a Metric of Language Comprehension Difficulty. Journal of Cognitive Science, 9(2):159–191. ↗
  23. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  24. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. find our card
  25. Julius Steuer, Marie-Pauline Krielke, Stefan Fischer, Stefania Degaetano-Ortlieb, Marius Mosbach, and Dietrich Klakow. 2024. Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal. In Proceedings of the 17th Workshop on Building and Using Comparable Corpora (BUCC)@ LREC-COLING 2024, pages 12–23. ↗
  26. Elke Teich, Peter Fankhauser, Stefania Degaetano-Ortlieb, and Yuri Bizzoni. 2021. Less is More/-More Diverse: On the Communicative Utility of Linguistic Conventionalization. Frontiers in Communication, 5. ↗
  27. Gui Wang, Hui Wang, Xinyi Sun, Nan Wang, and Li Wang. 2023. Linguistic complexity in scientific writing: A large-scale diachronic study from 1821 to 1920. Scientometrics, 128(1):441–460. ↗
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find our card
  29. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. Publisher: arXiv Version Number: 4. 11. find
  30. Language Resource References Davies, Mark. 2021. Corpus of Historical American English (COHA). UCLA Dataverse. find
  31. Fischer, Stefan and Knappen, Jörg and Menzel, Katrin and Teich, Elke. 2020. The Royal Society Corpus 6.0: Providing 300+ Years of Scientific Writing for Humanistic Study. European Language Resources Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/steuer-2026-modeling-memory-surprisal-trade-off/