Elamur Research⌕
☰
home / papers / 2021 / Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Paper · 2021 · Peter Hase, Mona Diab, Asli Celikyilmaz et al.

Summary

The paper asks whether language models hold anything resembling beliefs, and how such beliefs can be measured, corrected and inspected. The authors adopt Dennett's minimal view of belief and test ~100M-parameter models fine-tuned on FEVER, zsRE and LeapOfThought. They check whether answers stay stable across paraphrases and whether they respect entailments, including contrapositives. They also build logically unrelated "local neutral" pairs from Wikidata5m to verify that an edit stays contained.

For editing, they extend the learned optimizer KnowledgeEditor (De Cao et al., 2021) with SLAG, an objective whose name stands for Sequential, Local, and Generalizing and which trains the editor to make updates with all three of those properties. SLAG trains on harder data and applies several small edit steps. The authors argue that editors should be judged on repairing wrong beliefs, not on swapping one plausible answer for another. They also introduce belief graphs, in which an edge means that editing one belief flips another.

Models showed only weak belief-like behaviour. Well-tuned standard optimizers were strong baselines for single edits, but SLAG won when many beliefs were changed one after another. The work provides evaluation tools for safer post-training correction of models.

Key points

  • Models of about 100M parameters agree with themselves across paraphrases less than 70% of the time, and their consistency under entailment is uneven.
  • KnowledgeEditor nearly breaks down when beliefs are updated sequentially, and off-the-shelf optimizers beat it in that setting. Training with SLAG lets the learned optimizer surpass these baselines again.
  • The FEVER belief graph has 10,444 nodes and 1.88m edges, while the LeapOfThought graph has 8,642 nodes and 9.71m edges. In both graphs every node has at least one edge (0% edgeless), and a few hub beliefs affect hundreds of others.
  • Update-Transitivity is 66.64% on FEVER and 24.38% on LeapOfThought, far from the 100% a logically coherent model would show. At the 95th percentile, a single edit corrupts 211 (FEVER) and 2,752 (LeapOfThought) previously correct predictions.
  • Edits tend to flip beliefs the model already got wrong. The code is released at github.com/peterbhase/SLAG-Belief-Updating.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States77 works · 23,365 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal and Srinivasan Iyer. 2021. Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs. arXiv:2111.13654.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.13654 · arXiv 2111.13654

@misc{hase2021do,
  title = {{Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs}},
  author = {Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal and Srinivasan Iyer},
  year = {2021},
  doi = {10.48550/arxiv.2111.13654},
  eprint = {2111.13654},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.13654}
}

References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  2. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696. find
  3. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In EMNLP, pages 6491–6506. Association for Computational Linguistics. find
  4. Daniel Dennett. 1995. Do animals have beliefs? Comparative approaches to cognitive science, 111. find
  5. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021. Time-aware language models as temporal knowledge bases. arXiv preprint arXiv:2106.15110.
  6. Bradley Efron and Robert J Tibshirani. 1994. An Introduction to the Bootstrap. CRC press. find
  7. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. find
  8. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP.
  9. Anastasia Giannakidou and Alda Mari. 2020. A linguistic framework for knowledge, belief, and veridicality judgement. HAL. find
  10. Peter Hase and Mohit Bansal. 2021. When can models learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201. find
  11. Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1772– 1791, Online… find
  12. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In EMNLP. find
  13. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  14. Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021. Beliefbank: Adding memory to a pre-trained language model for a systematic notion of belief. arXiv preprint arXiv:2109.14723.
  15. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Sebastian Ruder, Dani Yogatama, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. In NeurIPS. find
  16. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
  17. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS.
  18. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
  19. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. find
  20. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.11309. find
  21. Albert Newen and Tobias Starzak. 2020. How to ascribe beliefs to animals. Mind & Language. find
  22. German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71.
  23. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  24. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. find
  25. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866.
  26. Eric Schwitzgebel. 2019. Belief. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Fall 2019 edition. Metaphysics Research Lab, Stanford University. find
  27. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In ICLR. find
  28. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge. In NeurIPS.
  29. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  30. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In EMNLP.
  31. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of ACL.
  32. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194. find
  33. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178.
  34. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hase-2021-language-models-have-beliefs-methods/