Elamur Research⌕
☰
home / papers / 2022 / Language Models as Agent Models

Language Models as Agent Models

Paper · 2022 · Jacob Andreas

Summary

This position paper asks whether language models trained only on text can capture anything about the goals and mental states behind human language use. A common objection holds that they cannot. Andreas offers a narrower claim. To predict the next token, an LM may build rough, incomplete models of what the likely author of the context, and any agents it mentions, believe, want and intend. These inferred states then causally shape what the model generates.

The argument has three parts:

  • Toy experiment: an LSTM is trained on a synthetic "encyclopedia" written by authors with conflicting belief sets.
  • Formal framing: the Belief–Desire–Intention model is used to define what such agent states mean.
  • Literature survey: case studies from existing work, illustrated with GPT-3 samples, show both successes and failures.

The toy model recovers author identity and can be steered to imitate a particular author. This shows that globally incoherent data can still produce locally coherent agent simulation. The paper argues that text-only pre-training is a useful base for building intentional, goal-directed language agents. However, new architectures are needed for memory, planning and grounding.

Key points

  • Toy setup: 10,000 synthetic documents, each 10 propositions long, written by A-type, B-type or O-type authors. The O-type authors hold contradictory beliefs. The model is a 512-dimensional LSTM, trained with no author labels.
  • Author-consistent samples: 31% of generated documents matched A-type authors, 33% matched B-type authors, and 36% matched only O-type authors.
  • Probing and control: a linear probe on the hidden state at the 5th token identified the author with 98% accuracy. Setting the initial hidden state to the average A-type representation made 89% of outputs A-type.
  • Examples and framework: illustrative samples come from GPT-3 text-davinci-002 with greedy decoding. The formal framing uses Bratman's BDI model.
  • Limitations:
    • All the evidence is English-only.
    • Fixed-depth networks cannot carry out planning-style search.
    • Scratchpads use up context that is needed to describe the simulated agent.
    • Inference failures can produce untruthful answers, for example on TruthfulQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Jacob Andreas. 2022. Language Models as Agent Models. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.423

@inproceedings{andreas2022language,
  title = {{Language Models as Agent Models}},
  author = {Jacob Andreas},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.423},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.423}
}

References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Langshaw Austin. 1975. How to do things with words. Oxford University Press. find
  2. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. find
  3. Emily M Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
  4. Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2020. Energy-based reranking: Improving neural machine translation using energy-based models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  5. Yonatan Bisk, Christos Christodoulopoulos, and Julia Hockenmaier. 2015. Labeled grammar induction with minimal supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. find
  6. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020. Experience grounds language. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
  7. Paul Bloom. 2002. How children learn the meanings of words. MIT Press. find
  8. Michael Bratman. 1987. Intention, plans, and practical reason. University of Chicago Press. find
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems.
  10. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. E-SNLI: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems.
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  12. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  13. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2018. Universal transformers. In Proceedings of the International Conference on Learning Representations. find
  14. Daniel Dennett. 1995. Do animals have beliefs? Comparative approaches to cognitive science. find
  15. Daniel Clement Dennett. 1987. The intentional stance. MIT press. find
  16. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
  17. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. Advances in Neural Information Processing Systems. find
  18. Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. 2022. Semantic projection recovers rich human knowledge of multiple object features from word embeddings. Nature Human Behaviour. find
  19. H Paul Grice. 1969. Utterer’s meaning and intentions. The Philosophical Review. find
  20. Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654. find
  21. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks. In Proceedings of the International Conference on Learning Representations. find
  22. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the Conference the North American Chapter of the Association for Computational Linguistics.
  23. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Associat…
  25. Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  26. Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R Gormley, and Jason Eisner. 2020. Limitations of autoregressive models and their alternatives. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
  27. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics.
  28. Gary Marcus and Ernest Davis. 2020. GPT-3, bloviator: OpenAI’s language generator has no idea what it’s talking about. Technology Review. find
  29. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. arXiv preprint arXiv:2202.05262. find
  30. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  31. Lalchand Pandia and Allyson Ettinger. 2021. Sorting through the noise: Testing robustness of information processing in pre-trained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  32. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444.
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
  34. Sebastian Schuster and Tal Linzen. 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
  35. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press. find
  36. Michael Tomasello. 2005. Constructing a language: A usage-based theory of language acquisition. Harvard University Press. find
  37. Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum. 2021. Human-level reinforcement learning through theory-based modeling, exploration, and planning. arXiv preprint arXiv:2107.12544. — [arXiv:2107.12544](https://arxiv.org/abs/2107.12544) find
  38. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In ACM Conference on Fairness, Accountability, and Transparency. find
  39. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in Neural Information Processing Systems.
  40. Rolf A Zwaan and Carol J Madden. 2005. Embodied sentence comprehension. Grounding cognition: The role of perception and action in memory, language, and thinking. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/andreas-2022-language-models-as-agent-models/