Elamur Research⌕
☰
home / papers / 2024 / Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal

Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal

Paper · 2024 · Julius Steuer, Marie-Pauline Krielke, Stefan Fischer et al.

Summary

The paper asks how to measure diachronic change in English scientific writing with surprisal from transformer language models instead of the 4-gram models used in earlier work. N-gram models only see three preceding words, so they cannot capture structures that span longer stretches, such as relative clauses. The authors use the Royal Society Corpus (RSC), which covers 1665–1996. Training comparable models on it is hard because the number of texts, tokens per text and vocabulary all grow exponentially over time, so early periods have little data.

To handle this, they use one vocabulary shared across all periods. Each model begins with training on an even sample drawn from the entire corpus, after which pre-training continues on one particular time segment. They compare the resulting surprisal trends with earlier n-gram findings and run a focused study of relative clauses, adding dependency length annotations.

Transformer surprisal proved to be a useful predictor, especially for complex structures. Wider context combined with dependency length gives a richer view of communicative efficiency. The recipe should carry over to other low-resource settings.

Key points

  • The data is the Royal Society Corpus (Philosophical Transactions and Proceedings of the Royal Society of London): 295,895,749 tokens in 47,837 texts, published 1665–1996.
  • RSC annotations include TreeTagger lemmas and POS tags and Universal Dependency parses produced by Stanza. The texts were OCR'd and cleaned beforehand.
  • Earlier studies used 4-gram surprisal and found that grammatical patterns become more conventionalized over time, for example the noun–preposition–noun pattern such as oxide of iron.
  • Training strategy:
    • a shared vocabulary across all periods;
    • initial training on uniformly sampled data from the whole corpus;
    • continued pre-training on individual temporal segments.
  • The case study targets relative clauses, which depend on long-range context, and combines transformer surprisal with dependency length.

Interesting ideas

  • I1. One vocabulary is fixed across all corpus slices so that surprisal values stay comparable even when slice sizes and lexicons differ wildly.
  • I2. The models start from a base trained on a uniform sample of the whole corpus and then specialize by continued training on each segment, which counters the bias toward the data-rich periods.
  • I3. For cognitive plausibility, training data should not include text from after the period being modeled, since a speaker of that time could not have read it.
  • I4. LM surprisal is paired with dependency length from UD parses to study one construction where long context matters (relative clauses).
  • I5. Instead of judging the model only by global averages, the authors probe one construction known to need long context.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Keeping one lexicon and tokenizer inventory across genres (fables, school texts, word problems) would make per-genre unknown-word and ambiguity statistics directly comparable.
  • I2 (learn): Train the gparse or tagger base on a genre-balanced mix and then fine-tune per register, so small registers are not drowned out by large silver sets.
  • I3 (learn): The same no-future-data principle can serve as a leakage check, making sure evaluation texts and their sources never enter silver or teacher training data.
  • I4 (tree): Combine surprisal with arc length to flag parses whose relative-clause attachments look improbable, as candidates for review or repair rules.
  • I5 (answer): Build targeted test sets around specific constructions such as relative clauses, so that errors in readers' answers can be traced to the structures that cause them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksHistorical linguistics & language change · 249 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Julius Steuer, Marie-Pauline Krielke, Stefan Fischer, Stefania Degaetano-Ortlieb, Marius Mosbach and Dietrich Klakow. 2024. Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal. Proceedings of the 17th Workshop on Building and Using Comparable Corpora (BUCC) @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{steuer2024modeling,
  title = {{Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal}},
  author = {Julius Steuer, Marie-Pauline Krielke, Stefan Fischer, Stefania Degaetano-Ortlieb, Marius Mosbach and Dietrich Klakow},
  booktitle = {Proceedings of the 17th Workshop on Building and Using Comparable Corpora (BUCC) @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/3ubzf7o8hspf},
  url = {https://doi.org/10.63317/3ubzf7o8hspf}
}

References

58 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation and Compiling, volume 1. Prentice-Hall, Englewood Cliffs, NJ. find
  2. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. ↗
  3. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L 1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. ↗
  4. Douglas Biber and Bethany Gray. 2011. Grammatical change in the noun phrase: the influence of written language use. 15(2):223–250. ↗
  5. Douglas Biber and Bethany Gray. 2016. Grammatical Complexity in Academic English: Linguistic Change in Writing. Studies in English Language. Cambridge University Press. find
  6. Douglas Biber, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 1999. Longman grammar of spoken and written English. Longman. find
  7. Ashok K. Chandra, Dexter C. Kozen, and Larry J. Stockmeyer. 1981. Alternation. Journal of the Association for Computing Machinery, 28(1):114–133. find
  8. James W. Cooley and John W. Tukey. 1965. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation, 19(90):297–301. ↗
  9. Stefania Degaetano-Ortlieb. 2021. The rise of compounds as informationally dense structures in 20th-century scientific English: Chapter 11. In Elena Seoane measuring informativity. and Douglas Biber, editors, Corpus-based Approaches to Register Variation, Studies in Corpus Linguistics, pages 291–312. John Benjamins Publishing Company. find
  10. Stefania Degaetano-Ortlieb, Hannah Kermes, Ashraf Khamis, and Elke Teich. 2018. An Information-Theoretic Approach to Modeling Diachronic Change in Scientific English. Brill. Pages: 258-281 Section: From Data to Evidence in English Language Research. ↗
  11. Stefania Degaetano-Ortlieb and Elke Teich. 2016. Information-based modeling of diachronic linguistic change: from typicality to productivity. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 165–173. find
  12. Stefania Degaetano-Ortlieb and Elke Teich. 2018. Using relative entropy for detection and analysis of periods of diachronic linguistic change. In Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 22–33, Santa Fe, New Mexico. Association for Computational Linguistics. ↗
  13. Stefania Degaetano-Ortlieb and Elke Teich. 2019. Toward an optimal code for communication: The case of scientific English. Corpus Linguistics and Linguistic Theory, 18(1):175–207. ↗
  14. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. 109(2):193– 210. ↗
  15. Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Timeout: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for Computational Linguistics. find
  16. Stefan Fischer, Jörg Knappen, Katrin Menzel, and Elke Teich. 2020. The Royal Society Corpus 6.0: Providing 300+ years of scientific writing for humanistic study. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 794–802, Marseille, France. European Language Resources Association. find
  17. Stefan L. Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in Cognitive Science, 5(3):475– 494. ↗
  18. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. 112(33):10336–10341. Publisher: National Academy of Sciences. ↗
  19. Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 446–457. Association for Computational Linguistics.
  20. Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK. ↗
  21. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, NAACL ’01, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics.
  22. John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive Science, 30(4):643–672. ↗
  23. M.A.K. Halliday. 1988. On the language of physical science. In Mohsen Ghadessy, editor, Registers of written English: Situational factors and linguistic features, pages 162–177. Pinter. Tex.date-added: 2010-03-09 11:19:11 +0100 tex.date-modified: 2010-03-30 15:16:26 +0200. find
  24. M.A.K. Halliday and James R. Martin. 1993. Writing science: Literacy and discursive power. Falmer Press. find
  25. William L. Hamilton, J. Leskovec, and Dan Jurafsky. 2016. Cultural shift or linguistic drift? comparing two computational measures of semantic change. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  26. Zellig Harris Sabbettai. 1991. A Theory of Language and Information: A Mathematical Approach. Oxford University Press, Oxford, New York. ↗
  27. Hans Jürgen Heringer, Bruno Strecker, and Rainer Wimmer. 1980. Syntax: Fragen, Lösungen, Alternativen. Fink. find
  28. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. ↗
  29. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc. find
  30. Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza, and Mariona Coll Ardanuy. 2021. Neural Language Models for Nineteenth-Century English. ArXiv:2105.11321 [cs]. ↗
  31. Richard Hudson. 1995. Measuring syntactic difficulty. Manuscript, University College, London. Marianne Hundt, David Denison, and Gerold Schneider. Relative complexity in scientific discourse. 16(2):209–240. Md. find
  32. Hussain and Ishtiak Mahmud. 2019. pymannkendall: a python package for non-parametric Mann Kendall family of trend tests. Journal of Open Source Software, 4(39):1556. find
  33. Tom S Juzek, Marie-Pauline Krielke, and Elke Teich. 2020. Exploring diachronic syntactic shifts with dependency length: the case of scientific English. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 109–119, Barcelona, Spain (Online). Association for Computational Linguistics.
  34. Hannah Kermes, Stefania Degaetano-Ortlieb, Ashraf Khamis, Jörg Knappen, and Elke Teich. 2016. The Royal Society Corpus: From uncharted data to corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1928–1931, Portorož, Slovenia. European Language Resources Association (ELRA). find
  35. Hannah Kermes and Elke Teich. 2017. Average Surprisal of Parts-of-(s)peech. Birmingham, UK. find
  36. Corpus Linguistics 2017. find
  37. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61–65, Baltimore, MD, USA. Association for Computational Linguistics.
  38. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 181–184, Detroit, MI, USA. IEEE. find
  39. Marie-Pauline Krielke. 2021. Relativizers as markers of grammatical complexity: A diachronic, cross-register study of English and German. Bergen Language and Linguistics Studies, 11(1):91–120. ↗
  40. Marie-Pauline Krielke, Luigi Talamo, Mahmoud Fawzi, and Jörg Knappen. 2022. Tracing syntactic change in the scientific genre: Two Universal Dependency-parsed diachronic corpora of scientific English and German. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4808–4816, Marseille, France. European Language Resources Association. ↗
  41. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. ↗
  42. Katrin Menzel, Jörg Knappen, and Elke Teich. 2021. Generating Linguistically Relevant Metadata for the Royal Society Corpus. ↗
  43. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An Analysis of Neural Language Modeling at Multiple Scales. Publisher: arXiv Version Number: 1. ↗
  44. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukas Burget, and Jan Honza Cernocky. 2011. RNNLM - Recurrent Neural Network Language Modeling Toolkit. In IEEE Automatic Speech Recognition and Understanding Workshop. Edition: IEEE Automatic Speech Recognition and Understanding Workshop. find
  45. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗
  46. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  47. Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version 2. ↗
  48. Simone Scaboro, Beatrice Portelli, Emmanuele Chersoni, Enrico Santus, and Giuseppe Serra. 2021. NADE: A benchmark for robust adverse drug events extraction in face of negations. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 230–237, Online. Association for Computational Linguistics. find
  49. Hans-Jörg Schmid. 2015. A blueprint of the Entrenchment-and-Conventionalization model. 3(1):3–26. Publisher: De Gruyter Mouton. ↗
  50. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Language Processing, Manchester, UK. find
  51. Claude E. Shannon. 1948. A Mathematical Theory of Communication. Bell System Technical Journal, 27(3):379–423. ↗
  52. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In 7th International Conference on Spoken Language Processing (IC-SLP 2002), pages 901–904. ISCA. find
  53. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746–3753, Istanbul, Turkey. European Language Resource Association (ELRA).
  54. Elke Teich, Peter Fankhauser, Stefania Degaetano-Ortlieb, and Yuri Bizzoni. 2021. Less is more/more diverse: On the communicative utility of linguistic conventionalization. 5:142. ↗
  55. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. ↗
  56. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. 8.
  57. Language Resource References Fischer, Stefan and Knappen, Jörg and Menzel, Katrin and Teich, Elke. 2020. The Royal Society Corpus 6.0: Providing 300+ Years of Scientific Writing for Humanistic Study. European Language Resources Association. [link]. find
  58. Kermes, Hannah and Degaetano-Ortlieb, Stefania and Khamis, Ashraf and Knappen, Jörg and Teich, Elke. 2016. The Royal Society Corpus: From Uncharted Data to Corpus. European Language Resources Association (ELRA). [link]. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/steuer-2024-modeling-diachronic-change-english-scientific/