Elamur Research⌕
☰
home / papers / 2023 / Testing the Predictions of Surprisal Theory in 11 Languages

Testing the Predictions of Surprisal Theory in 11 Languages

Summary

Evidence for surprisal theory, the idea that words harder to predict from prior context take longer to read, comes mostly from English readers. This paper tests whether the theory holds across typologically diverse languages. The authors use the MECO dataset, which provides eye-tracking records for comparable texts in many languages, and analyze eleven of them spanning five families. Word-level surprisal and contextual entropy (expected surprisal) come from autoregressive language models: the multilingual mGPT, plus monolingual models trained on large corpora and on a size-matched small corpus of about 30 million words per language. Linear regression models of reading times are compared by how much a target predictor raises held-out log-likelihood over a baseline (delta log-likelihood). Surprisal improves prediction in every language, and contextual entropy adds further power in most of them. Linear models fit as well as more flexible ones, which supports a linear link. As the broadest cross-linguistic test so far, the study strengthens the tie between information theory and incremental reading.

Key points

  • The study tests three hypotheses: surprisal predicts reading times, contextual entropy predicts reading times, and the surprisal–reading time relation is linear. All three are supported cross-linguistically.
  • The eleven languages are Korean, Turkish, Hebrew and Finnish, along with seven others: Spanish, Russian, Italian, Greek, German, English and Dutch. Estonian from MECO was excluded because mGPT was not trained on it.
  • The multilingual mGPT and the monolingual models showed no clear difference in predicting reading times. Even the smallest monolingual models were trained on about 30M tokens, so truly low-resource settings remain untested.
  • The authors contrast their approach with concurrent MECO work by de Varda and Marelli (2022). That study used the masked model mBERT, which sees context on both sides, so its pseudo-surprisal estimates are not psychologically plausible.
  • The authors list several limitations. Indo-European languages dominate the sample, and each other family has only one language. All languages are high-resource, and the readers come from industrialized societies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00612

@article{wilcox2023testing,
  title = {{Testing the Predictions of Surprisal Theory in 11 Languages}},
  author = {Wilcox, Ethan G. and Pimentel, Tiago and Meister, Clara and Cotterell, Ryan and Levy, Roger P.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00612},
  url = {https://doi.org/10.1162/tacl_a_00612}
}

References

65 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: The case for Basque. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4781–4788, Marseille, Fran… find
  2. Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. https://doi.org/10.1016/j.jml.2012.11.001, PubMed: 24403724 ↗
  3. Veronica Boyce, Richard Futrell, and Roger P. Levy. 2020. Maze made easy: Better and easier measurement of incremental processing difficulty. Journal of Memory and Language, 111:104082. https://doi.org/10.1016/j.jml.2019.104082 ↗
  4. Veronica Boyce and Roger Levy. 2020. A-maze of natural stories: Texts are comprehensible using the maze task. In Talk at 26th Architectures and Mechanisms for Language Processing conference (AMLaP 26). Potsdam, Germany. find
  5. Trevor Brothers and Gina R. Kuperberg. 2021. Word predictability effects are linear, not logarithmic: Implications for probabilistic models of sentence comprehension. Journal of Memory and Language, 116:104174. https://doi.org/10.1016/j.jml.2020.104174, PubMed: 33100508 — [doi:10.1016/j.jml.2020.104174](https://doi.org/10.1016/j.jml.2020.104174) ↗
  6. Benedetta Cevoli, Chris Watkins, and Kathleen Rastle. 2022. Prediction as a basis for skilled reading: Insights from modern language models. Royal Society Open Science, 9(6):211837. https://doi.org/10.1098/rsos.211837, PubMed: 35719885 — [doi:10.1098/rsos.211837](https://doi.org/10.1098/rsos.211837) ↗
  7. Charles Clifton, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. Eye Movements, pages 341–371. https://doi.org/10.1016/B978-008044980 -7/50017-3 — [doi:10.1016/b978-008044980](https://doi.org/10.1016/b978-008044980) ↗
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  9. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages… find
  10. Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances, 5(9):1–10. https://doi.org/10.1126/sciadv .aaw2594, PubMed: 32047854 — [doi:10.1126/sciadv](https://doi.org/10.1126/sciadv) ↗
  11. Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non-native reading. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 138–144, Online only. Association for Computational Linguistics. find
  12. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. https://doi.org/10 .1016/j.cognition.2008.07.008, PubMed: 18930455 find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  14. Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2021. A primer on pretrained multilingual language models. arXiv preprint arXiv:2107.00676. — [arXiv:2107.00676](https://arxiv.org/abs/2107.00676) find
  15. Kenneth I. Forster, Christine Guerrera, and Lisa Elliot. 2009. The maze task: Measuring forced incremental sentence processing time. Behavior Research Methods, 41(1):163–171. https://doi.org/10.3758/BRM.41.1 .163, PubMed: 19182136 — [doi:10.3758/brm.41.1](https://doi.org/10.3758/brm.41.1) ↗
  16. Victoria Fossum and Roger Levy. 2012. Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012), pages 61–69, Montréal, Canada. Association for Computational Linguistics. find
  17. Austin F. Frank and T. Florian Jaeger. 2008. Speaking rationally: Uniform information density as an optimal strategy for language production. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 30. find
  18. Stefan Frank. 2010. Uncertainty reduction as a measure of cognitive processing effort. In Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Linguistics, pages 81–89, Uppsala, Sweden. Association for Computational Linguistics. find
  19. Stefan L. Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in Cognitive Science, 5(3):475–494. https://doi.org/10.1111/tops.12025, PubMed: 23681508 ↗
  20. Stefan L. Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829–834. https://doi.org/10.1177/0956797611409589, PubMed: 21586764 — [doi:10.1177/0956797611409589](https://doi.org/10.1177/0956797611409589) ↗
  21. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput… find our card
  22. W. J. Granger. 1969. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3):424–438. https://doi.org/10.2307/1912791 ↗
  23. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440–2452, Marseille, France. European Language Resources Association. find
  24. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. https://doi.org/10.3115/1073336 .1073357 ↗ our card
  25. John Hale. 2003. The information conveyed by words in sentences. Journal of Psycholinguistic Research, 32:101–123. https://doi.org/10.1023/A:1022492123056, PubMed: 12690827 — [doi:10.1023/a:1022492123056](https://doi.org/10.1023/a:1022492123056) ↗
  26. John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive Science, 30(4). https://doi.org/10.1207/s15516709cog0000 64, PubMed: 21702829 — [doi:10.1207/s15516709cog0000](https://doi.org/10.1207/s15516709cog0000) ↗
  27. Betty Hart and Todd R. Risley. 1995. Meaningful Differences in the Everyday Experience of Young American Children. Paul H. Brookes Publishing Co, Baltimore, MD. find
  28. Martin Haspelmath, Matthew S. Dryer, David Gil, and Bernard Comrie. 2005. The World Atlas of Language Structures. Oxford University Press. find
  29. Dieter Hillert. 1998. Sentence Processing: A Crosslinguistic Perspective. Brill. https://doi.org/10.1163/9780585492230 — [doi:10.1163/9780585492230](https://doi.org/10.1163/9780585492230) ↗
  30. Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, and Lisa Beinborn. 2021. Multilingual language models predict human reading behavior. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1… find
  31. Jacob Louis Hoover, Morgan Sonderegger, Steven T. Piantadosi, and Timothy J. O’Donnell. 2022. The plausibility of sampling as an algorithmic theory of sentence processing. PsyArXiv preprint. https://doi.org/10 .31234/osf.io/qjnpv Jill Jegerski. 2013. Self-paced reading. In Research Methods in Second… — [doi:10.4324/9780203123430](https://doi.org/10.4324/9780203123430) find
  32. Marcel A. Just, Patricia A. Carpenter, and Jacqueline D. Woolley. 1982. Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2):228. https://doi.org/10.1037/0096-3445.111.2.228 ↗
  33. Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee corpus. In Proceedings of the 12th European Conference on Eye Movements. find
  34. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  35. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… find
  36. Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421–10436, Abu Dhabi, United Arab Emirates. Association for Comp… find
  37. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  38. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. https://doi.org/10.1016/j.cognition .2007.05.006, PubMed: 17662975 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) ↗
  39. Roger Levy and T. Florian Jaeger. 2006. Speakers optimize information density through syntactic reduction. Advances in Neural Information Processing Systems, 19. https://doi.org/10.7551/mitpress/7503.003 .0111 — [doi:10.7551/mitpress/7503.003](https://doi.org/10.7551/mitpress/7503.003) ↗
  40. Tal Linzen and T. Florian Jaeger. 2016. Uncertainty and expectation in sentence processing: Evidence from subcategorization distributions. Cognitive Science, 40(6):1382–1411. https://doi.org/10.1111/cogs.12274, PubMed: 26286681 — [doi:10.1111/cogs.12274](https://doi.org/10.1111/cogs.12274) ↗
  41. Steven G. Luke and Kiel Christianson. 2018. The Provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50:826–833. https://doi.org/10.3758/s13428-017-0908-4, PubMed: 28523601 — [doi:10.3758/s13428-017-0908-4](https://doi.org/10.3758/s13428-017-0908-4) ↗
  42. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963–980, Online and Punta Cana, Dominican Republic… find
  43. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic… find
  44. Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. https://doi.org/10.1162/tacl a 00548 ↗ our card
  45. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  46. François Pellegrino, Christophe Coupé, and Egidio Marsico. 2011. A cross-language perspective on speech information rate. Language, 87(3):539–558. https://doi.org/10.1353/lan.2011.0057 ↗
  47. Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 949–962, Online and Punta… find
  48. Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Roger Levy, and Ryan Cotterell. 2023. On the effect of anticipation on reading times. Transactions of the Association for Computational Linguistics. find
  49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):1–67. find
  50. Keith Rayner. 1975. The perceptual span and peripheral cues in reading. Cognitive Psychology, 7(1):65–81. https://doi.org/10.1016 /0010-0285(75)90005-5 find
  51. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3):372. https://doi.org/10.1037/0033-2909 .124.3.372, PubMed: 9849112 — [doi:10.1037/0033-2909](https://doi.org/10.1037/0033-2909) ↗
  52. Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009. Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 324–33… find
  53. Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019. Is multilingual BERT fluent in language generation? In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pages 29–36, Turku, Finland. Linköping University Electronic Press. find
  54. Elizabeth R. Schotter, Bernhard Angele, and Keith Rayner. 2012. Parafoveal processing in reading. Attention, Perception, & Psychophysics, 74(1):5–35. https://doi.org/10.3758/s13414-011-0219-2, PubMed: 22042596 ↗
  55. Cory Shain. 2019. A large-scale study of the effects of word frequency and predictability in naturalistic reading. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page… find
  56. Cory Shain. 2021. CDRNN: Discovering complex dynamics in human language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3718–3734, Online.… find
  57. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2022. Large-scale evidence for logarithmic effects of word predictability on reading time. PsyArXiv preprint. https://doi.org/10.31234/osf.io/4hyna Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bel… ↗ our card
  58. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, Sara Maria Da Fonseca, Nicolas Dirix, Wouter Duyck, Argyro Fella, Ram Frost, Carolina A. Gattei, Areti Kalaitzi, Nayoung Kwon, Kaidi… find
  59. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. https://doi.org/10.1016/j.cognition .2013.02.013, PubMed: 23747651 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) ↗
  60. Robyn Speer. 2022. rspeer/wordfreq: v3.0. find
  61. Marten van Schijndel and William Schuler. 2017. Approximations of predictive entropy correlate with reading times. In Proceedings of the Cognitive Science Society, pages 1260–1265. find
  62. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076. find our card
  63. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. find
  64. In Proceedings of the 2020 Meeting of the Cognitive Science Society, pages 1707–1713, Online. Cognitive Science Society. find
  65. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/