Elamur Research⌕
☰
home / papers / 2023 / Entity Tracking in Language Models

Entity Tracking in Language Models

Paper · 2023 · Najoung Kim and Sebastian Schuster

Summary

Understanding a discourse requires following how entities change as a text unfolds, but whether large language models can do this had only been tested indirectly. Kim and Schuster first revisit the probing study of Li et al. (2021) on Alchemy. They show that its reported success comes almost entirely from trivial cases that need no real tracking. They then build a controlled behavioral task. Synthetic English texts describe boxes holding objects, followed by a sequence of operations that put objects in, take them out, or move them. The model must then name the contents of a box. The data are generated from templates so that lexical cues, pretraining priors and slot-filling shortcuts cannot reveal the answer. Without finetuning, only GPT-3.5 models, which were trained on both text and code, perform non-trivially. GPT-3 and Flan-T5 do not. A finetuned T5 learns the task and partly generalizes to longer operation sequences and to objects unseen in training, though accuracy falls on harder splits. The authors conclude that text-only pretraining does not make entity tracking surface. The task also offers a tool for studying world models in LMs.

Key points

  • Reanalysis of Li et al. (2021): 62.7% of probed Alchemy beaker states equal the initial state, and about 87.6% of cases are trivial overall. The probe reaches 86.8% on trivial cases but only 3.1% on non-trivial ones.
  • Dataset design: 7 boxes, at most 3 objects per box, about 2 objects per box initially. It uses 100 object nouns drawn from the British National Corpus (BNC) with frequency above 27. There are 2200 random scenarios, each with 12 operations, and no "empty the box" operation.
  • Four desiderata guide evaluation: no pretraining-like state distributions, no single-word cues, low lexical overlap between training and test, and no solution by slot-filling.
  • Models evaluated: GPT-3, GPT-3.5 and Flan-T5 without finetuning, plus a finetuned T5. The authors speculate that code pretraining helps because programs require tracking variable states.
  • Limitations: evidence is purely behavioral, performance is unstable when many operations affect one entity, and only English was tested. Code and data are available at github.com/sebschu/entity-tracking-lms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany149 works · 18,349 citations🎓Boston UniversityBoston · United States30 works · 4,343 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Najoung Kim and Sebastian Schuster. 2023. Entity Tracking in Language Models. arXiv:2305.02363.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.02363 · arXiv 2305.02363

@misc{kim2023entity,
  title = {{Entity Tracking in Language Models}},
  author = {Najoung Kim and Sebastian Schuster},
  year = {2023},
  doi = {10.48550/arxiv.2305.02363},
  eprint = {2305.02363},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.02363}
}

References

58 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Aina, Carina Silberer, Ionut-Teodor Sorodoc, Matthijs Westera, and Gemma Boleda. 2019. What do entity-centric models learn? insights from entity linking in multi-party dialogue. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics… find
  2. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 44–54, Marseille, France. European Language Resources Association.
  3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  4. Ben Bogin. 2022. How does GPT-3 perform on COVR-10 splits with in-context learning? Twitter thread. find
  5. Ben Bogin, Shivanshu Gupta, and Jonathan Berant. 2022. Unobserved local structures make compositional generalization hard. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Abu Dhabi, United Arab Emirates. Association for Computational Linguistic… find
  6. Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018. Simulating action dynamics with neural process networks. In International Conference on Learning Representations (ICLR 2018).
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… — Language Models are Few-Shot Learners
  8. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172– 181, Brussels, Belgium. Association for Computationa…
  9. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  10. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… find
  11. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2019. Textworld: A learning environment for text-based games. In Computer Games, pages 41–75, Cham. Springer Internatio…
  12. Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, and Peter Clark. 2019. Everything happens for a reason: Discovering the purpose of actions in procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confere… find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Jeroen Groenendijk and Martin Stokhof. 1991. Dynamic predicate logic. Linguistics and Philosophy, 14:39–100. find
  15. Aditya Gupta and Greg Durrett. 2019a. Effective use of transformer networks for entity tracking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 759–769, Hong Kon… find
  16. Aditya Gupta and Greg Durrett. 2019b. Tracking discrete and continuous entity state for process understanding. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 7–12, Minneapolis, Minnesota. Association for Computational Linguistics. find
  17. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 385–393, Los Angeles, California. Association for Computatio…
  18. Irene Heim. 2002. File change semantics and the familiarity theory of definiteness. In Formal semantics: The essential readings, chapter 9, pages 223–248. Blackwell Publishing Oxford. find
  19. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. In International Conference on Learning Representations (ICLR 2017).
  20. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR 2016).
  21. Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. 2017. Dynamic entity representations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1830–1839, Copenhagen, Denmark. Association for Computational…
  22. Hans Kamp, Josef Van Genabith, and Uwe Reyle. 2011. Discourse Representation Theory, pages 125–394. Springer Netherlands, Dordrecht. find
  23. Lauri Karttunen. 1976. Discourse referents. In J. D. Mc-Cawley, editor, Syntax and Semantics Vol. 7, pages 363–386. Academic Press. find
  24. Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. 2015. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 982–992, Lisbon, Portugal. Association for Computatio… find
  25. Andrew Kyle Lampinen. 2022. Can language models handle recursively nested grammatical structures? a case study on comparing models and humans. arXiv:2210.15303.
  26. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 28–…
  27. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  28. Geoffrey Leech, Paul. Rayson, and Andrew Wilson. 2001. Word frequencies in written and spoken English: based on the British National Corpus. Longman. find
  29. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  30. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… find
  31. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Emergent world representations: Exploring a sequence model trained on a synthetic task. In International Conference on Learning Representations (ICLR 2023). find
  32. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210– 5217, Online. Association for Computational Linguistics.
  33. Sharid Loáiciga, Anne Beyer, and David Schlangen. 2022. New or old? exploring how pre-trained language models represent discourse entities. In Proceedings of the 29th International Conference on Computational Linguistics, pages 875–886, Gyeongju, Republic of Korea. International Committee on Computa… find
  34. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456–1465, Berlin, Germany. Association for Computational L…
  35. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384–1403, Abu Dhabi, United Arab Emirates. Association for Computatio…
  36. William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047– 1060.
  37. Mante S. Nieuwland and Jos J. A. Van Berkum. 2006. When peanuts fall in love: N400 evidence for the power of discourse. Journal of Cognitive Neuroscience, 18(7):1098–1111. find
  38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155.
  39. Christopher Potts. 2020. Is it possible for language models to achieve language understanding? Medium blog post. find
  40. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti…
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67.
  43. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association…
  44. Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, Abu Dhabi, United Arab Emirates. Association for Compu… find
  45. Sebastian Schuster and Tal Linzen. 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… find
  46. Ionut-Teodor Sorodoc, Kristina Gulordava, and Gemma Boleda. 2020. Probing for referential information in language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4177–4189, Online. Association for Computational Linguistics. find
  47. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
  48. Shubham Toshniwal, Sam Wiseman, Allyson Ettinger, Karen Livescu, and Kevin Gimpel. 2020. Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8519–8526, Online. Asso…
  49. Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2022. Chess as a testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385–11393. find
  50. Chen Feng Tsai, Xiaochen Zhou, Sierra S Liu, Jing Li, Mo Yu, and Hongyuan Mei. 2023. Can large language models play text games well? current state-of-the-art and open questions. arXiv:2304.02868. — [arXiv:2304.02868](https://arxiv.org/abs/2304.02868) find
  51. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the arrau corpus. Natural Language Engineering, 26(1):95–128. find
  52. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  53. ACE 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia: LDC2006T06. find
  54. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards AI-complete question answering: a set of prerequisite toy tasks. arXiv:1502.05698.
  55. Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Ph.D. thesis, Massachusetts Institute of Technology. find
  56. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  57. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics.
  58. A Additional Analyses of Results from Li et al. (2021) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2023-entity-tracking-language-models/