Elamur Research⌕
☰
home / papers / 2026 / An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal

An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal

Summary

Neural language models (LMs) estimate reading difficulty well on simple, naturalistic text. On garden-path sentences, however, their surprisal predicts far less slowdown than humans actually show. Several researchers have taken this gap as evidence that syntactic disambiguation cost lies outside surprisal theory. Yoshida et al. test the alternative explanation: perhaps current LMs simply predict words differently from people.

Building on the reading-time alignment approach of Kiegeland et al. (2024), they fine-tune off-the-shelf LMs on garden-path items annotated with self-paced reading times. The goal is for surprisal-based reading-time estimates to match human data. A regularization term limits drift during training, and evaluation uses leave-one-out cross-validation.

The fine-tuned models reproduce human slowdowns on held-out garden-path items without overfitting. They also predict naturalistic reading times better than before and keep their general LM abilities. This is an existence proof that surprisal alone can account for both kinds of data. The authors argue that such flexibility threatens the falsifiability of surprisal theory, and they suggest ways to constrain it.

Key points

  • Earlier work (Huang et al., 2024) found that LM surprisal accounts for only about 1/10 to 1/30 of human self-paced reading slowdowns on garden-path sentences.
  • The training data cover three garden-path constructions (MVRR, NPS, NPZ), each with ambiguous and unambiguous conditions and annotated regions of interest. The authors describe this as the largest such dataset with human reading times known to them.
  • Models fine-tuned on a single construction generalize to the other, unseen garden-path constructions. They fail, however, to capture the SRC/ORC asymmetry, a difficulty that is probably better explained by memory-based theories.
  • The authors propose two routes to a falsifiable surprisal theory:
    • restricting how the probability distribution may be obtained, for example by limiting fine-tuning or requiring human-scale corpora;
    • demanding that the posited parse distribution be psychologically real.
  • Code is released at github.com/osekilab/RE-GPE. Stated limitations are the small dataset, coverage of English only, and no analysis of the models' internal mechanisms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Language models & pretraining · Psycholinguistics & language acquisition

Institutions

🎓The University of TokyoTokyo · Japan31 works · 7 442 citations🔬National Institute for Japanese Language and LinguisticsTokyo · Japan4 works · 2 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1694

@inproceedings{yoshida2026existence,
  title = {{An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal}},
  author = {Yoshida, Ryo and Isono, Shinnosuke and Someya, Taiga and Oseki, Yohei and Kuribayashi, Tatsuki},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1694},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1694}
}

References

45 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 301–313, Abu Dhabi, United… find
  2. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. In Cognition and the Development of Language, pages 279–362. Wiley, New York. find
  3. Jeffrey S. Bowers and Jeff Mitchell. 2025. Studies with impossible languages falsify LMs as models of human language. Preprint, arXiv:2511.11389. — [arXiv:2511.11389](https://arxiv.org/abs/2511.11389) find
  4. Noam Chomsky. 1965. Aspects of the Theory of Syntax. The MIT Press, Cambridge. find
  5. Janet Dean Fodor and Fernanda Ferreira. 1998. Reanalysis in Sentence Processing. Studies in Theoretical Psycholinguistics ; v.21. Kluwer Academic Publishers, Dordrecht ;. find
  6. Stefan L. Frank, Irene Fernandez Monsalve, Robin L. Thompson, and Gabriella Vigliocco. 2013. Reading time data for evaluating broad-coverage models of English sentence processing. Behavior Research Methods, 45(4):1182–1190. find
  7. Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. 2015. The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140:1–11. find
  8. Karl Friston. 2010. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2):127–138. find
  9. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The Natural Stories Corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language… find
  10. Edward Gibson. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium, pages 94–126. The MIT Press, Cambridge, MA, US. find
  11. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput… find our card
  12. Daniel Grodner and Edward Gibson. 2005. Consequences of the Serial Nature of Linguistic Input for Sentenial Complexity. Cognitive Science, 29(2):261– 290. find
  13. John Hale. 2001. A Probabilistic Earley Parser as a Psycholinguistic Model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  14. John T. Hale. 2014. Automaton Theories of Human Sentence Comprehension. CSLI Studies in Computational Linguistics. CSLI Publications, Stanford, CA. find
  15. Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137:104510. find
  16. Shinnosuke Isono. 2024. Category Locality Theory: A unified account of locality effects in sentence comprehension. Cognition, 247:105766. find
  17. Madhur Mangalam. 2025. The myth of the Bayesian brain. European Journal of Applied Physiology, 125(10):2643–2677. find
  18. Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychological Review, 87:329–354. find
  19. David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman and Company, San Francisco. find
  20. Marcel A. Just, Patricia A. Carpenter, and Jacqueline D. Woolley. 1982. Paradigms and processes in reading comprehension. Journal of Experimental Psychology: General, 111(2):228–238. find
  21. Samuel Kiegeland, Ethan Wilcox, Afra Amini, David Robert Reich, and Ryan Cotterell. 2024. Reverse-Engineering the Reader. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9367–9389, Miami, Florida, USA. Association for Computational Linguistics. find
  22. D. C. Mitchell. 1984. An Evaluation of Subject-Paced Reading Tasks and Other Methods for Investigating Immediate Processes in Reading 1. In New Methods in Reading Comprehension Research. Routledge. find
  23. Byung-Doh Oh and William Schuler. 2023. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics, 11:336–350. find our card
  24. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. Psychometric Predictive Power of Large Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1983–2005, Mexico City, Mexico. Association for Computational Linguistics. find
  25. Byung-Doh Oh and William Schuler. 2024. Leading Whitespaces of Language Models’ Subword Vocabulary Pose a Confound for Calculating Word Probabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472, Miami, Florida, USA. Association for Comp… find
  26. Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context Limitations Make Neural Language Models More Human-Like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421–10436, Abu Dhabi, United Arab Emirates. Association for Comp… find
  27. Tiago Pimentel and Clara Meister. 2024. How to Compute the Probability of a Word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. find
  28. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  29. Roger Levy. 2013. Memory and surprisal in human sentence comprehension. In Sentence Processing, Current Issues in the Psychology of Language, pages 78–114. Psychology Press, New York, NY, US. find
  30. Roger Levy and Edward Gibson. 2013. Surprisal, the PDC, and the primary locus of processing difficulty in relative clauses. Frontiers in Psychology, 4. find
  31. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. find
  32. Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Roger P. Levy, and Ryan Cotterell. 2023. On the Effect of Anticipation on Reading Times. Transactions of the Association for Computational Linguistics, 11:1624–1642. find
  33. Karl R. Popper. 1959. The Logic of Scientific Discovery. Routledge, London. find
  34. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  35. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  36. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  37. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  38. Robyn Speer. 2022. Rspeer/wordfreq: V3.0. Zenodo. find
  39. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. find
  40. Adrian Staub. 2025. Predictability in Language Comprehension: Prospects and Problems for Surprisal. Annual Review of Linguistics, 11(Volume 11, 2025):17–34. find
  41. Wilson L. Taylor. 1953. “Cloze Procedure”: A New Tool for Measuring Readability. Journalism Quarterly, 30(4):415–433. find
  42. William Timkey, Kuan-Jung Huang, Byung-Doh Oh, Grusha Prasad, Suhas Arehalli, Tal Linzen, and Brian Dillon. 2025. Eye movements reveal a dissociation between prediction and structural processing in language comprehension. find
  43. Marten van Schijndel and Tal Linzen. 2021. Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty. Cognitive Science, 45(6):e12988. find
  44. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. In Proceedings of the… find
  45. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yoshida-2026-existence-proof-neural-language-models/