Elamur Research⌕
☰
home / papers / 2026 / Transformer Attention as a Unified Model of Encoding and Retrieval in Human Sentence Processing

Transformer Attention as a Unified Model of Encoding and Retrieval in Human Sentence Processing

Summary

Memory-based theories of sentence comprehension aim to explain both how linguistic items are stored and how they are later accessed. However, leading models such as ACT-R formalize retrieval in detail and leave encoding largely unspecified. Earlier work argued that Transformer attention acts as a cue-based memory, but it was tested mainly on syntactic retrieval effects.

This paper asks whether attention also reflects semantic interference that arises at encoding. It builds on the finding by Gordon et al. that object relative clauses become less difficult when the head noun and the embedded noun belong to different referential classes. For example, a definite description can be paired with a name, a pronoun or a quantified pronoun instead of another description.

GPT-2 attention was analyzed with three measures: Earth Mover's Distance, attention-to-target and attention entropy. Each measure reproduced the human behavioral pattern across the referential manipulations. The author concludes that attention covers interference at both encoding and retrieval, on syntactic and semantic dimensions. This gives it broader coverage than ACT-R and supports a single computational account of memory in sentence processing.

Key points

  • The model is GPT-2, and the three attention-based metrics are Earth Mover's Distance, attention-to-target and attention entropy. All three matched human data on referential-class effects in relative clauses.
  • The test case is the ORC/SRC asymmetry. The extra difficulty of object relatives shrinks when the two nouns differ in referential class (common noun vs. name, pronoun or quantified pronoun), following Gordon et al. (2001, 2004).
  • Referential class is unlikely to act as a retrieval cue for the embedded verb. The effect is therefore treated as encoding interference, which the literature calls "feature overwriting" (Nairne; Oberauer and Kliegl).
  • The author argues that keys and queries come from the same learned projections. On this view, encoding and retrieval share one representational geometry: similar nouns are placed close together in key space, and attention is then split between them at the verb.
  • The study extends Ryu and Lewis (2021, 2025), whose work was limited to retrieval effects. It appeared at CMCL @ LREC 2026, pages 1–9.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5mv4jv4mi7em

@inproceedings{parker2026transformer,
  title = {{Transformer Attention as a Unified Model of Encoding and Retrieval in Human Sentence Processing}},
  author = {Parker, Dan},
  booktitle = {Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2026},
  doi = {10.63317/5mv4jv4mi7em},
  url = {https://doi.org/10.63317/5mv4jv4mi7em}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Richard Futrell, Edward Gibson, and Roger P. Levy. 2021. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44:1–54. find
  2. Andrew Gelman, Daniel Simpson, and Michael Betancourt. 2017. The prior can often only be understood in the context of the likelihood. Entropy, 19(555):1–13. find
  3. Edward Gibson. 2000. Dependency locality theory: A distance-based theory of linguistic complexity. In Alec Marantz, Yasushi Miyashita, and Wayne O’Neil, editors, Image, language, brain: Papers from the first mind articulation project symposium, pages 95–126. MIT Press. find
  4. Peter C. Gordon, Randall Hendrick, and Marcus Johnson. 2001. Memory interference during language processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(6):1411–1423. find
  5. Peter C. Gordon, Randall Hendrick, and Marcus Johnson. 2004. Effects of noun phrase type on sentence complexity. Journal of Memory and Language, 51:97–104. find
  6. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision, 40:99–121. find
  7. Daniel Grodner and Edward Gibson. 2005. Consequences of the serial nature of linguistic input for sentenial complexity. Cognitive Science, 29:261– 290. find
  8. Soo Hyun Ryu and Richard L. Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61–71… find
  9. Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. 2019. Exbert: A visual analysis tool to explore learned representations in transformers models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational… find
  10. Soo Hyun Ryu and Richard L. Lewis. 2025. Memory for prediction: A transformer-based theory of sentence processing. Journal of Memory and Language, 145:1–19. find
  11. Cassandra L. Jacobs and Maryellen C. MacDonald. 2024. Constraint satisfaction in large language models. Language, Cognition and Neuroscience, 39(10):1231–1248. find
  12. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2018. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference… find our card
  13. Jonathan King and Marcel A. Just. 1991. Individual differences in syntactic processing: The role of working memory. Journal of Memory and Language, 30:580–602. find
  14. Richard L. Lewis and Shravan Vasishth. 2005. An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29(3):375–419. find
  15. Julie A. Van Dyke and Clinton L. Johns. 2012. Memory interference as a determinant of language comprehension. Language and Linguistics Compass, 6(4):193–211. find
  16. Richard L. Lewis, Shravan Vasishth, and Julie A. Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in Cognitive Sciences, 10(10):447–454. find
  17. Julie A. Van Dyke and Brian McElree. 2006. Retrieval interference in sentence comprehension. Journal of Memory and Language, 55:157–166. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS 2017). find
  19. James S. Nairne. 1990. A feature model of immediate memory. Memory & Cognition, 18:251–269. find
  20. Klaus Oberauer and Reinhold Kliegl. 2006. A formal model of capacity limits in working memory. Journal of Memory and Language, 55:601–626. find
  21. Julia El Zini and Mariette Awad. 2022. On the explainability of natural language processing deep models. ACM Computing Surveys, 55(5):1–31. find
  22. Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5(777963). find
  23. Byung-Doh Oh and William Schuler. 2022. Entropy-and distance-based predictors from gpt-2 attention patterns predict reading times over and above gpt-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324–9334. Association for Computational… find
  24. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parker-2026-transformer-attention-as-unified-model/