Elamur Research⌕
☰
home / papers / 2026 / Aggregated Transformer Attention Measures Predict Reading Times Beyond Surprisal

Aggregated Transformer Attention Measures Predict Reading Times Beyond Surprisal

Summary

Surprisal from transformer language models predicts human reading times well, but memory-related processing difficulty may need other signals. This paper asks whether measures built from self-attention add explanatory power beyond surprisal. The authors define four attention-based cost measures, each tied to a memory theory of sentence processing: entropy (similarity-based interference), expected backward distance (Dependency Locality Theory), a recency-weighted activation score (ACT-R-style decay), and a memory-focus difference between consecutive positions. Each measure is computed in two ways. In the head-averaged ("local") version, it is computed for every head and then averaged. In the "global" version, it is computed once from attention pooled over all heads. Both versions are also tested with and without scaling by context length. The authors train a next-token-prediction transformer on WikiText. They then fit linear mixed-effects models of go-past times in the Provo eye-tracking corpus, with frequency, length, surprisal and spillover terms as the baseline. Only the unscaled head-averaged measures reliably improve fit over surprisal. The global and length-scaled variants help little or not at all. The results suggest that aggregation choices largely determine whether attention-based costs are behaviourally useful.

Key points

  • Four memory-motivated attention metrics are evaluated in one framework: entropy, distance, activation and attention difference. Distance has only one version, because head-averaging and global pooling give the same expected value for it.
  • Following Oh and Schuler (2022), attention weights are made norm-based by including value vectors and per-head output projections. Mixed-effects models with likelihood ratio tests are applied to go-past times from the Provo Corpus.
  • Much of the predictive signal of the local metrics overlaps with a word's position in the sentence. This links them to the growing memory demands of an expanding context. The globally aggregated metrics are almost collinear with each other (Pearson r ≈ 0.95–1.00).
  • The language model was trained on WikiText (about 3.57M training sentences of 3–60 words). The checkpoint with the lowest validation loss, at about 8.8 epochs, was used. Training on 4 Nvidia A100 GPUs took about 10.5 hours.
  • As a sanity check, attention distance was compared with dependency lengths from the UD English Web Treebank. When dependency length was counted as the number of intervening nouns and verbs, the correlation was only weak (Pearson r = 0.13).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4xn5igbwvtd5

@inproceedings{mielczarek2026aggregated,
  title = {{Aggregated Transformer Attention Measures Predict Reading Times Beyond Surprisal}},
  author = {Mielczarek, Lukas and Kallmeyer, Laura},
  booktitle = {Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2026},
  doi = {10.63317/4xn5igbwvtd5},
  url = {https://doi.org/10.63317/4xn5igbwvtd5}
}

References

47 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John R. Anderson, Daniel Bothell, Michael D. Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. 2004. An integrated theory of the mind. Psychological review, 111 4:1036–60. find
  2. Suhas Arehalli, Brian Dillon, and Tal Linzen. 2023. Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities. ArXiv:2210.12187 [cs]. find
  3. Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. 2022. Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1105– 1119. Association for Computational Linguistics. find
  4. Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255– 278. find
  5. Douglas Bates, Reinhold Kliegl, Shravan Vasishth, and Harald Baayen. 2018. Parsimonious mixed models. find
  6. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1–48. find our card
  7. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300. find
  8. Robert W. Booth and Ulrich W. Weger. 2013. The function of regressions in reading: Backward eye movements allow rereading. MEMORY & COGNITION, 41(1):82–97. find
  9. Kenneth P. Burnham and David R. Anderson. 1998. Model Selection and Inference. A Practical Information-Theoretic Approach, first edition. Springer Books. Springer, Newy York, New York, USA. find
  10. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… find
  11. Kate Ehrlich and Keith Rayner. 1983. Pronoun assignment and semantic integration during reading: eye movements and immediacy of processing. Journal of Verbal Learning and Verbal Behavior, 22(1):75–87. find
  12. Michael A. A. Eskenazi. 2024. Best practices for cleaning eye movement data in reading research. Behavior Research Methods, 56(3):2083–2093. find
  13. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal archive, 12:23– 38. find
  14. Edward Gibson. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium, pages 94–126. The MIT Press, Cambridge, Massachusetts, USA. find
  15. Peter C. Gordon, Randall Hendrick, Marcus Johnson, and Yoonhyoung Lee. 2006. Similarity-based interference during language comprehension: Evidence from eye tracking during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(6):1304–1321. find
  16. Peter C. Gordon, Randall Hendrick, and William H. Levine. 2002. Memory-Load Interference in Syntactic Processing. Psychological Science, 13(5):425–430. find
  17. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In NAACL ’01: Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001, pages 1–8, Morristown, New Jersey, USA. Association for Computational Linguistics. find our card
  18. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational… find our card
  19. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  20. Richard L. Lewis. 1993. An architecturally-based theory of human sentence comprehension. Ph.D. thesis, CARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF COMPUTER SCI-ENCE. find
  21. Richard L. Lewis. 1996. Interference in short-term memory: The magical number two (or three) in sentence processing. Journal of Psycholinguistic Research, 25(1):93–115. find
  22. Richard L. Lewis and Shravan Vasishth. 2005. An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29(3):375–419. find
  23. Richard L. Lewis, Shravan Vasishth, and Julie A. Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in cognitive sciences, 10(10):447–454. find
  24. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826–833. find
  25. Emma Marsden, Sophie Thompson, and Luke Plonsky. 2018. A methodological synthesis of self-paced reading in second language research. Applied Psycholinguistics, 39(5):861–904. find
  26. Lukas Mielczarek, Timothée Bernard, Laura Kallmeyer, Katharina Spalek, and Benoit Crabbé. 2025. Modelling expectation-based and memory-based predictors of human reading times with syntax-guided attention. In Proceedings of the Second Workshop on the Bridges and Gaps between Formal and Computational… find
  27. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, Volume 5 - 2022. find
  28. Byung-Doh Oh and William Schuler. 2022. Entropy-and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal. ArXiv:2212.11185 [cs]. find
  29. Ralph Radach, Lynn Huestegge, and Ronan Reilly. 2008. The role of global top-down factors in local eye-movement control in reading. PSYCHO-LOGICAL RESEARCH-PSYCHOLOGISCHE FORSCHUNG, 72(6):675–688. find
  30. Ralph Radach and Alan Kennedy. 2013. Eye movements in reading: Some theoretical context. QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 66(3, SI):429–452. find
  31. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  32. K Rayner. 1998. Eye movements in reading and information processing: 20 years of research. PSYCHOLOGICAL BULLETIN, 124(3):372–422. find
  33. Soo Hyun Ryu and Richard L. Lewis. 2021. Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61–71… find
  34. Soo Hyun Ryu and Richard L. Lewis. 2022. Using transformer language model to integrate surprisal, entropy, and working memory retrieval accounts of sentence processing. 35th Annual Conference on Human Sentence Processing. find
  35. Soo Hyun Ryu and Richard L. Lewis. 2025. Memory for prediction: A Transformer-based theory of sentence processing. Journal of Memory and Language, 145:104670. find
  36. Robyn Speer. 2022. rspeer/wordfreq: v3.0. [Online; accessed 26-January-2025]. find
  37. William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720, Singapore. Association for Computational Linguistics. find
  38. Julie A. Van Dyke and Richard L. Lewis. 2003. Distinguishing effects of structure and decay on attachment and repair: A cue-based parsing account of recovery from misanalyzed ambiguities. Journal of Memory and Language, 49(3):285– 316. find
  39. Merity, Stephen and Xiong, Caiming and Bradbury, James and Socher, Richard. 2016. WikiText-103: A Large Language Modeling Corpus. Salesforce Research. find
  40. Natalia Silveira and Timothy Dozat and Marie-Catherine de Marneffe and Samuel Bowman and Miriam Connor and John Bauer and Christopher D. Manning. 2014. Universal Dependencies English Web Treebank. Universal Dependencies. find
  41. Marten van Schijndel and Tal Linzen. 2021. Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty. Cognitive Science, 45(6):e12988. find
  42. Shravan Vasishth and Richard L. Lewis. 2004. Modeling sentence processing in ACT-R. In Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together, IncrementParsing ’04, pages 82–87, USA. Association for Computational Linguistics. find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  44. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I… find our card
  45. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470. find
  46. Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The Linearity of the Effect of Surprisal on Reading Times across Languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721, Singapore. Association for Computational Linguistics. 11. find
  47. Language Resource References Luke, Steven G. and Christianson, Kiel. 2018. The Provo Corpus. Open Science Framework. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mielczarek-2026-aggregated-transformer-attention-measures-predict/