Elamur Research⌕
☰
home / papers / 2026 / Memory efficiency and resource-rational encoding in sentence processing

Memory efficiency and resource-rational encoding in sentence processing

Paper · 2026 · Weijie Xu, Brian Dillon and Richard Futrell

Summary

Language models used as models of human reading usually have far more memory for context than people do. The authors propose a resource-rational way to limit working memory (WM) in Transformers. Instead of restricting what attention can reach, they make stored information less precise. They add Gaussian noise to the self-attention value vectors of a GPT-2 small model. The noise precision shrinks exponentially as the distance between query and key grows, and each attention head learns its own decay rate. Training on wikitext-103 uses a hybrid objective that combines next-word prediction with an explicit penalty on total encoding precision, weighted by λ. Two main findings emerge:

  • Better fit to reading times. Constrained models' surprisal predicts human reading times better than unconstrained models.
  • Compressed predictions. Tighter constraints make next-word predictions more compressed and categorical, with coarser distinctions between contexts.

This offers a testable prediction for psycholinguistics. It also separates how memory is retrieved from how memories are represented, and it links WM limits to quantization in engineering.

Key points

  • Noise model: the precision of value vector k seen from position q is τ₀·e^(−θ(q−k)), and the decay rate θ is trained separately for each head.
  • Lower-layer attention heads learned faster precision decay than higher layers.
  • Fit to reading times follows a Goldilocks pattern. It peaks at a moderate constraint, for example λ = 0.01 on the SPR NSC corpus, rather than improving steadily as λ increases.
  • Setup: GPT-2 small trained on English wikitext-103. Decay depends only on linear distance, so it ignores hierarchical structure.
  • Suggested future work includes crosslinguistic tests, other model sizes, and State Space Models for truly incremental processing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations🎓University of California, IrvineIrvine · United States43 works · 36,565 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 108 works

Cite

Cite as Weijie Xu, Brian Dillon and Richard Futrell. 2026. Memory efficiency and resource-rational encoding in sentence processing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1550

@inproceedings{xu2026memory,
  title = {{Memory efficiency and resource-rational encoding in sentence processing}},
  author = {Weijie Xu, Brian Dillon and Richard Futrell},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1550},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1550}
}

References

87 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. George A Alvarez. 2011. Representing multiple objects as an ensemble enhances visual cognition. Trends in Cognitive Sciences, 15(3):122–131. find
  2. John Robert Anderson. 1990. The Adaptive Character of Thought. Psychology Press. find
  3. Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638–14663.
  4. R Harald Baayen, Douglas J Davidson, and Douglas M Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. find
  5. Brian Bartek, Richard L Lewis, Shravan Vasishth, and Mason R Smith. 2011. In search of on-line locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5):1178. find
  6. Christopher J Bates and Robert A Jacobs. 2020. Efficient data compression in perception and perceptual memory. Psychological Review, 127(5):891. find
  7. Paul M Bays, Raquel FG Catalao, and Masud Husain. 2009. The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10):7–7. find
  8. Paul M Bays, Sebastian Schneegans, Wei Ji Ma, and Timothy F Brady. 2024. Representation and computation in visual working memory. Nature Human Behaviour, pages 1–19. find
  9. Veronica Boyce, Richard Futrell, and Roger P Levy. 2020. Maze made easy: Better and easier measurement of incremental processing difficulty. Journal of Memory and Language, 111:104082. find
  10. Veronica Boyce and Roger Levy. 2023. A-maze of Natural Stories: Comprehension and surprisal in the Maze task. Glossa Psycholinguistics, 2(1). find
  11. Timothy F Brady, Maria M Robinson, and Jamal R Williams. 2024. Noisy and hierarchical visual memory across timescales. Nature Reviews Psychology, pages 1–17. find
  12. Timothy F Brady and Joshua B Tenenbaum. 2013. A probabilistic model of visual working memory: Incorporating higher order regularities into working memory capacity estimates. Psychological Review, 120(1):85. find
  13. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3):430–441. find
  14. Noam Chomsky. 1957. Syntactic structures. Mouton de Gruyter. find
  15. Christian Clark, Byung-Doh Oh, and William Schuler. 2025. Linear recency bias during training improves transformers’ fit to reading times. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7735–7747. find
  16. Charles Clifton Jr, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. Eye movements, pages 341–371. find
  17. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, pages 10041–10071. find
  18. Andrea De Varda and Marco Marelli. 2024. Locally biased transformers better align with human reading times. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30– 36. find
  19. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318– 30332. find
  20. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. find
  21. Stanley Dubinsky and Kemp Williams. 1995. Recategorization of prepositions as complementizers: The case of temporal prepositions in english. Linguistic Inquiry, 26(1):125–137. find
  22. Jeffrey L Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  23. Michael C Frank and Noah D Goodman. 2025. Cognitive modeling using artificial intelligence. Annual Review of Psychology, 77. find
  24. Stefan L Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829–834. find
  25. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. find
  26. Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814. find
  27. Richard Futrell, Edward Gibson, Harry J Tily, Idan Blank, Anastasia Vishnevetsky, Steven T Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55:63–77. find
  28. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. arXiv preprint arXiv:2501.17047. — [arXiv:2501.17047](https://arxiv.org/abs/2501.17047) find
  29. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245):273–278. find
  30. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76. find
  31. Edward Gibson, Neal Pearlmutter, Enriqueta Canseco-Gonzalez, and Gregory Hickok. 1996. Recency preference in the human sentence processing mechanism. Cognition, 59(1):23–59. find
  32. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, and 1 others. 2022. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369–380. find
  33. Laura Gwilliams, Alec Marantz, David Poeppel, and Jean-Rémi King. 2025. Hierarchical dynamic coding coordinates speech comprehension in the human brain. Proceedings of the National Academy of Sciences, 122(42):e2422097122. find
  34. Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal. Psychological Review, 128(4):726. find
  35. Michael Hahn and Richard Futrell. 2019. Estimating predictive rate–distortion curves via neural variational inference. Entropy, 21(7):640. find
  36. Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119. find
  37. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  38. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  39. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1725–1744. find
  40. Andrej Karpathy. 2023. nanoGPT. https://github.com/karpathy/nanoGPT. ↗
  41. Maayan Keshev, Mandy Cartner, Aya Meltzer-Asscher, and Brian Dillon. 2025. A working memory model of sentence processing as binding morphemes to syntactic positions. Topics in Cognitive Science, 17(1):88– 105. find
  42. Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421–10436. find
  43. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large language models are human-like internally. Transactions of the Association for Computational Linguistics, 13:1743–1766.
  44. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  45. Richard L Lewis. 1996. Interference in short-term memory: The magical number two (or three) in sentence processing. Journal of Psycholinguistic Research, 25(1):93–115. find
  46. Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in Cognitive Science, 6(2):279–311. find
  47. Richard L Lewis and Shravan Vasishth. 2005. An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29:375– 419. find
  48. Richard L Lewis, Shravan Vasishth, and Julie A Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in Cognitive Sciences, 10(10):447–454. find
  49. Falk Lieder and Thomas L Griffiths. 2020. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43:e1. find
  50. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pages 87–100. find
  51. Steven G Luke and Kiel Christianson. 2018. The provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833. find
  52. Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. 2006. Bayesian inference with probabilistic population codes. Nature Neuroscience, 9(11):1432–1438. find
  53. Wei Ji Ma, Masud Husain, and Paul M Bays. 2014. Changing concepts of working memory. Nature Neuroscience, 17(3):347–356. find
  54. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences. find
  55. David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Henry Holt and Co., Inc., New York, NY, USA. find
  56. Brian McElree. 2006. Accessing recent events. Psychology of Learning and Motivation, 46:155–200. find
  57. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  58. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):81. find
  59. Masato Mita, Ryo Yoshida, and Yohei Oseki. 2025. Developmentally-plausible working memory shapes a critical period for language acquisition. arXiv preprint arXiv:2502.04795.
  60. Dennis Norris. 2006. The Bayesian reader: Explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113(2):327. find
  61. Klaus Oberauer. 2009. Design for a working memory. Psychology of Learning and Motivation, 51:45–100. find
  62. Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. arXiv preprint arXiv:2510.05141. — [arXiv:2510.05141](https://arxiv.org/abs/2510.05141) find
  63. Byung-Doh Oh and William Schuler. 2022. Entropy-and distance-based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324–9334. find
  64. Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350.
  65. William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720. find
  66. John C Trueswell, Michael K Tanenhaus, and Susan M Garnsey. 1994. Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33(3):285–318. find
  67. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  68. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
  69. Tamar I Regev, Colton Casto, Eghbal A Hosseini, Markus Adamek, Anthony L Ritaccio, Jon T Willie, Peter Brunner, and Evelina Fedorenko. 2024. Neural populations in the language network differ in the size of their temporal receptive windows. Nature Human Behaviour, 8(10):1924–1942. find
  70. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy.
  71. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65. find
  72. Soo Hyun Ryu and Richard L Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61– 71… find
  73. Soo Hyun Ryu and Richard L Lewis. 2025. Memory for prediction: A transformer-based theory of sentence processing. Journal of Memory and Language, 145:104670. find
  74. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and 1 others. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyL… find
  75. Ethan G Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.
  76. Jacqueline S Sachs. 1974. Memory in reading and listening to discourse. Memory & Cognition, 2(1):95–100. find
  77. Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721. find
  78. Elizabeth R Schotter and Brian Dillon. 2025. A beginner’s guide to eye tracking for psycholinguistic studies of reading. Behavior Research Methods, 57(2):68. find
  79. Weijie Xu and Richard Futrell. 2026. Strategic resource allocation in memory encoding: An efficiency principle shaping language processing. Journal of Memory and Language, 146:104706. find
  80. Herbert A Simon. 1955. A behavioral model of rational choice. The Quarterly Journal of Economics, pages 99–118. find
  81. Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7937–7942. find
  82. Garrett Smith and Shravan Vasishth. 2020. A principled approach to feature selection in models of sentence processing. Cognitive Science, 44(12):e12918. find
  83. Robyn Speer. 2022. rspeer/wordfreq: v3.0. find
  84. Junyuan Zhao, Ruimin Gao, and Jonathan R Brennan. 2025. Decoding the neural dynamics of headed syntactic structure building. Journal of Neuroscience, 45(17). find
  85. Susanne Still. 2014. Information bottleneck approach to predictive inference. Entropy, 16(2):968–989. find
  86. A James V Stone. 2018. Principles of neural information theory: Computational neuroscience and metabolic efficiency. Sebtel Press. find
  87. Training procedure. Models are trained on Wikitext-103 dataset (Merity et al., 2016) with 16000 training steps. The training procedure Model training with all items from the inanimate category (i.e., cross-category distance). Then, the separatability between the two categories of a linguistic featur… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-memory-efficiency-resource-rational-encoding/