Elamur Research⌕
☰
home / papers / 2026 / Human-like Fleeting Memory Improves Language Learning but Impairs Reading Time Prediction in Transformer Language Models

Human-like Fleeting Memory Improves Language Learning but Impairs Reading Time Prediction in Transformer Language Models

Summary

Cognitive science has long held that rapidly decaying working memory may help humans acquire language, a view rooted in Elman's 1993 connectionist simulations. Transformers seem to contradict this, since they learn syntax well with perfect recall of their whole context. Thamma and Heilbron test the idea directly by building "fleeting memory transformers." These models add a fixed, non-trainable power-law decay to the causal self-attention mask, so attention weakens with distance. They also include an "echoic" buffer that keeps the last few words fully intact. Matched models with and without this decay were trained on the developmentally plausible BabyLM data at two sizes, using paired seeds to isolate the effect.

With the buffer, fleeting memory reliably lowered validation loss and raised BLiMP accuracy. However, it made surprisal-based prediction of human reading times worse. Existing accounts of why stronger language models can fit reading times worse do not explain this gap. The results support memory limits as a helpful inductive bias for learning, but not as a route to more human-like behavioural predictions.

Key points

  • Training data: BabyLM sets of 10M and 100M words. Comparisons are paired runs sharing weight initialisation and data order, tested with bootstrap t-tests.
  • Approximate results (Figure 1, perfect vs. fleeting memory):
    • Validation loss: about 3.850 → 3.837 at 10M and 3.435 → 3.414 at 100M.
    • BLiMP: about 68.97 → 70.01 at 10M and 72.10 → 74.41 at 100M.
  • Role of the buffer: Decay alone did not help. Gains appeared only with the echoic buffer, which retains the initial 3–7 words perfectly. The buffer was motivated by human memory and set using the 10M data only.
  • Reading times: Fit to reading times (e.g., Dundee) got worse within each data scale. Superhuman training scale and memorisation of rare words do not explain the drop.
  • Relation to prior work: The findings contrast with De Varda and Marelli (2024) and Clark et al. (2025), who reported that recency biases improved reading time fit. Those studies used post-hoc fitted decay or ALiBi, unlike the fixed decay applied here during training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Language models & pretraining · Psycholinguistics & language acquisition

Institutions

🎓University of AmsterdamAmsterdam · Netherlands71 works · 79 805 citations🔬Max Planck Institute for PsycholinguisticsNijmegen · Netherlands5 works · 11 991 citations🎓Vrije Universiteit AmsterdamAmsterdam · Netherlands31 works · 3 248 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.688

@article{thamma2026human,
  title = {{Human-like Fleeting Memory Improves Language Learning but Impairs Reading Time Prediction in Transformer Language Models}},
  author = {Thamma, Abishek and Heilbron, Micha},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.688},
  url = {https://doi.org/10.1162/tacl.a.688}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislas Dehaene. 2021. How We Learn: Why Brains Learn Better Than Any Machine... for Now. Penguin. paradoxical impairment in reading time prediction reveals a surprising distinction: a model that is more human-like in its architectural constraints is not necessarily more human-like in its predicti… find
  2. Tatsuya Aoyama and Ethan Wilcox. 2025. Language models grow less humanlike beyond phase transition. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24938–24958, Vienna, Austria. Association for Computational Linguistics. find
  3. Chris Donkin and Robert M. Nosofsky. 2012. A power-law model of psychological memory strength in short- and long-term recognition. Psychological Science, 23(6):625–634. find
  4. Jeffrey L. Elman. 1993. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71–99. find
  5. Kristijan Armeni, Christopher Honey, and Tal Linzen. 2022. Characterizing verbatim short-term memory in neural language models. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 405–424, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computati… find
  6. Richard Futrell, Edward Gibson, and Roger P. Levy. 2020a. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814. find
  7. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55(1):63–77. find
  8. Alan Baddeley. 1992. Working memory. Science, 255(5044):556–559. find
  9. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1–48. find our card
  10. Richard Futrell, Roger P. Levy, and Edward Gibson. 2020b. Dependency locality as an explanatory principle for word order. Language, 96(2):371–412. find
  11. Marc Brysbaert and Boris New. 2009. Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4):977–990. find
  12. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, page 1–98. find
  13. Morten H. Christiansen and Nick Chater. 2016. The Now-or-Never bottleneck: A fundamental constraint on language. Behavioral and Brain Sciences, 39:e62. find
  14. Susan E. Gathercole, Catherine S. Willis, Hazel Emslie, and Alan D. Baddeley. 1992. Phonological memory and vocabulary development pages 923–929. European Language Resources Association (ELRA). during the early school years: A longitudinal study. Developmental Psychology, 28(5):887– 898. find
  15. Brian MacWhinney. 2004. MacWhinney corpus. find
  16. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18. Association for Computational Linguistics. find our card
  17. CHILDES English Masato Mita, Ryo Yoshida, and Yohei Oseki. 2025. Developmentally-plausible working memory shapes a critical period for language acquisition. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9386–9399, Vienna, Au… find our card
  18. Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119. find
  19. Elissa L. Newport. 1990. Maturational constraints on language learning. Cognitive Science, 14(1):11–28. find
  20. Peter M. C. Harrison, Roberta Bianco, Maria Chait, and Marcus T. Pearce. 2020. PPM-Decay: A computational model of auditory prediction with memory decay. PLOS Computational Biology, 16(11):1–41. find
  21. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5. Publisher: Frontiers. find
  22. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  23. Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. arXiv preprint arXiv:2510.05141v1. — [arXiv:2510.05141](https://arxiv.org/abs/2510.05141) find
  24. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… find
  25. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361v1. find our card
  26. Byung-Doh Oh and William Schuler. 2023b. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. find our card
  27. Andrej Karpathy. 2022. NanoGPT. https://github.com/karpathy/nanoGPT. ↗
  28. Henry W. Lin and Max Tegmark. 2017. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299. find
  29. Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (… find
  30. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(Volume 7, 2021):195–212. find
  31. Pierre Lison and Jörg Tiedemann. 2016. Open-Subtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), and Samuel R. Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs… find
  32. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  33. Ethan G Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 42. find
  34. Alec Radford, Jeff Wu, R. Child, D. Luan, Dario Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. find
  35. Guillaume Rousselet, Cyril R Pernet, and Rand R Wilcox. 2023. An introduction to the bootstrap: a versatile method to make inferences by using data-driven simulations. Meta-Psychology, 7. find
  36. Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2024. Using computational models to test syntactic learnability. Linguistic Inquiry, 55(4):805–848. find
  37. Caroline Rowland. 2013. Understanding Child Language Acquisition. Routledge. find
  38. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  39. William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720, Singapore. Association for Computational Linguistics. find
  40. Aditya Vaidya, Javier Turek, and Alexander Huth. 2023. Humans and language models diverge when predicting repeating text. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 58–69, Singapore. Association for Computational Linguistics. find
  41. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… find
  42. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thamma-2026-human-like-fleeting-memory-improves/