Elamur Research⌕
☰
home / papers / 2026 / Probing for Reading Times

Probing for Reading Times

Summary

The paper asks whether the hidden states of neural language models hold information about human reading behaviour beyond what the usual single-number predictor, surprisal, provides. The authors train regularized linear regression probes that map layer-wise representations onto three word-level eye-tracking measures: the duration of a word's first fixation, its gaze duration, and the overall time spent reading it. They use the Provo and MECO corpora, covering English, Greek, Hebrew, Russian and Turkish, with representations from mGPT, GPT-2 and cosmosGPT. Every layer is compared with three scalar baselines: surprisal, information value and logit-lens surprisal.

For early-pass measures, especially in English, representations from early layers predict reading times better than surprisal. This hints that model depth roughly mirrors the time course of human reading. For total reading time, compact surprisal is usually as good or better. Adding surprisal to the representations often helps. The best predictor depends heavily on the language and the measure. This matters for psycholinguistic modelling, because no single LM-derived feature works best in every setting.

Key points

  • Data: the Provo and MECO eye-tracking corpora, five languages (English, Greek, Hebrew, Russian, Turkish).
  • Models: mGPT (up to 1.3B parameters, the largest model tested), GPT-2 and cosmosGPT. Multi-token units are represented by mean-pooled hidden states.
  • Early-layer representations beat surprisal on first fixation and gaze duration. Scalar surprisal and logit-lens surprisal stay competitive or superior for total reading time, and often in the non-English languages.
  • Evaluation uses raw MSE with permutation testing to control for chance associations. Representations vary more across layers than information value or logit-lens surprisal do.
  • Code is available at github.com/rycolab/llm-representations-rt. Stated limitations include eye-tracking data only, no randomly initialized model baselines, and no dimensionality reduction (e.g., PCA).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Psycholinguistics & language acquisition Interpretability

 Institutions

🎓University College LondonLondon · United Kingdom44 works · 14,853 citations🎓Toyota Technological Institute at ChicagoChicago · United States22 works · 9,297 citations🎓ETH ZurichZurich · Switzerland24 works · 3,273 citations

Cite

Cite as Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak and Ryan Cotterell. 2026. Probing for Reading Times. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.575

@inproceedings{tsipidi2026probing,
  title = {{Probing for Reading Times}},
  author = {Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak and Ryan Cotterell},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.575},
  url = {https://doi.org/10.18653/v1/2026.acl-long.575}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024a. On the proper treatment of tokenization in psycholinguistics. In Proceedings of the 2024 Conf… find
  2. Mario Giulianelli, Andreas Opedal, and Ryan Cotterell. 2024b. Generalized measures of anticipation and responsivity in online language processing. In Findings of the Association for Computational Linguistics: EMNLP 2024. find
  3. Mario Giulianelli, Sarenne Wallbridge, Ryan Cotterell, and Raquel Fernández. 2026. Incremental alternative sampling as a lens into the temporal and representational resolution of linguistic prediction. Journal of Memory and Language, 148. find
  4. Mario Giulianelli, Sarenne Wallbridge, and Raquel Fernández. 2023. Information value: Measuring utterance predictability as distance from plausible alternatives. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  5. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. our card
  6. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Meeting of the North American Chapter of the Association for Computational Linguistics. our card
  7. Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. 2022. Probing as quantifying inductive bias. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  8. Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4). find
  9. H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Ege-men Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, and M. Fatih Amasyali. 2024. Introducing cosmosGPT: Monolingual training for Turkish language models. In International Conference on INnovations in Intelligent SysTems and Applications (INISTA). find
  10. Samuel Kiegeland, Vésteinn Snæbjarnarson, Tim Vieira, and Ryan Cotterell. 2026. On the proper treatment of units in surprisal theory. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  11. Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. In The International Conference on Learning Representations. find
  12. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. Psychometric predictive power of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024. find
  13. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large language models are human-like internally. Transactions of the Association for Computational Linguistics, 13. find
  14. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3). find
  15. Steven G. Luke and Kiel Christianson. 2018. The Provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50. find
  16. Clara Meister, Tiago Pimentel, Thomas Clark, Ryan Cotterell, and Roger Levy. 2022. Analyzing wrap-up effects through an information-theoretic lens. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find
  17. Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. 2025. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? find
  18. In The International Conference on Learning Representations. nostalgebraist. 2020. Interpreting GPT: The logit lens. find
  19. Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11. our card
  20. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  21. Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, and Ethan Wilcox. 2024. On the role of context in reading time prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  22. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  24. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3). find
  25. Keith Rayner. 2009. Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62(8). find
  26. Keith Rayner and Martin H. Fischer. 1996. Mindless reading revisited: Eye movements during reading and scanning are different. Perception & Psychophysics, 58(5). find
  27. Keith Rayner, Gretchen Kambe, and Susan A. Duffy. 2000. The effect of clause wrap-up on eye movements during reading. The Quarterly Journal of Experimental Psychology Section A, 53(4). find
  28. Francesco Ignazio Re, Andreas Opedal, Glib Manaiev, Mario Giulianelli, and Ryan Cotterell. 2025. A spatio-temporal point process for fine-grained modeling of reading behavior. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  29. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)… find
  30. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10). find
  31. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-shot learners go multilingual. Transactions of the Association for Computational Linguistics, 12. our card
  32. Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, and Daria Chernova. 2022. Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO). Behavior Research… find
  33. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3). find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. our card
  35. Tim Vieira, Benjamin Lebrun, Mario Giulianelli, Juan Luis Gastaldi, Brian Dusell, John Terilla, Timothy J. O’Donnell, and Ryan Cotterell. 2025. From language models over tokens to language models over characters. In Proceedings of the International Conference on Machine Learning, volume 267. find
  36. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas work in English? on the latent language of multilingual transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). our card
  37. Jennifer C. White, Tiago Pimentel, Naomi Saphra, and Ryan Cotterell. 2021. A non-linear structural probe. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  38. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11. our card
  39. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Cognitive Science Society. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsipidi-2026-probing-reading-times/