Elamur Research⌕
☰
home / papers / 2023 / Rigorously Assessing Natural Language Explanations of Neurons

Rigorously Assessing Natural Language Explanations of Neurons

Paper · 2023 · Jing Huang, Atticus Geiger, Karel D'Oosterlinck et al.

Summary

Natural-language descriptions of what individual neurons in large language models encode are appealing, but it is unclear whether they faithfully reflect model behavior. Huang et al. propose a two-part evaluation framework. In the observational mode, an explanation is treated as a prediction about which inputs should and should not trigger a neuron. Its meaning is approximated by a finite set of strings, and mismatches are counted as Type I errors (missed activations) or Type II errors (unexpected activations). In the intervention mode, the explanation is read as a causal claim. The authors build next-token prediction tasks that depend on the concept, then swap neuron values between inputs to check whether the output shifts as expected. They apply this to the GPT-4-written explanations of GPT-2 XL neurons from Bills et al. (2023). Even highly rated explanations frequently mispredict activations and show essentially no causal effect. The results caution against relying on such explanations for safety, bias or model-editing decisions. The authors argue instead for formal languages and distributed units of analysis.

Key points

  • The audit sampled 300 neurons (about 18%) from the roughly 1.7k GPT-2 XL neurons whose explanations Bills et al. scored at least 0.8 by GPT-4. Bills et al. explained about 300K neurons in total, and these high-scoring ones form only the top ~0.6%.
  • When used to predict activations, these explanations achieve 0.64 precision and 0.50 recall. The authors found no task where intervening on a single neuron causally changed model behavior.
  • Test data were generated with GPT-3.5-turbo: 20 in-concept words or phrases per explanation, each placed in sentences, plus corpus contexts where the neuron fires. Manual checks found over 95% of sentences valid. Spelling- and form-based explanations were checked with regex programs instead.
  • More than 30% of the Bills et al. explanations contain the vague phrase "related to." This illustrates the ambiguity and context dependence the authors see as inherent to natural-language explanations.
  • Dataset and code are released at github.com/explanare/eval_neuron_explanation. The paper appeared at the 6th BlackboxNLP Workshop (2023).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Ghent UniversityGhent · Belgium25 works · 2,949 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu and Christopher Potts. 2023. Rigorously Assessing Natural Language Explanations of Neurons. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.blackboxnlp-1.24

@inproceedings{huang2023rigorously,
  title = {{Rigorously Assessing Natural Language Explanations of Neurons}},
  author = {Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu and Christopher Potts},
  booktitle = {Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP},
  year = {2023},
  doi = {10.18653/v1/2023.blackboxnlp-1.24},
  url = {https://doi.org/10.18653/v1/2023.blackboxnlp-1.24}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. 2022. CE-Bab: Estimating the causal effects of real-world concepts on NLP model behavior. In Advances in Neural Information Processing Systems. find
  2. Omer Antverg and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations. find
  3. Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness tests for natural language explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), page…
  4. Our work contributes to improving the faithfulness of neuron interpretability methods that use natural language as a medium. Faithful explanation could provide the basis for safety assessments, bias detection efforts, model editing, and many other downstream applications. However, the ability to Yon… find
  5. We William Saunders. 2023. Language models can explain neurons in language models. emphasize that explanations of large language mod-https://openaipublic.blob.core.windows. els should always be used responsibly. net/neuron-explainer/paper/index.html. In an effort to evaluate the method proposed Adit… find
  6. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. find
  7. Virginie Do, Oana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. 2020. e-snli-ve-2.0: Corrected visual-textual entailment with natural language explanations. CoRR, abs/2004.03744. — [arXiv:2004.03744](https://arxiv.org/abs/2004.03744) find
  8. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer… find
  9. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386.
  10. Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems. find
  11. Atticus Geiger, Christopher Potts, and Thomas Icard. 2023a. Causal abstraction for faithful model interpretation. Ms., Stanford University. find
  12. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. Inducing causal structure for interpretable neural networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Lear… find
  13. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2023b. Finding alignments between interpretable causal variables and distributed neural representations. Ms., Stanford University. find
  14. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. find
  15. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Asso…
  16. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… find
  17. Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in… find
  18. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. 2022. Natural language descriptions of deep features. In International Conference on Learning Representations. find
  19. Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. 2023. Inducing character-level structure in subword-based language models with type-level interchange intervention training. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12163–12180, Toronto, Canada. Ass… find
  20. Ayush Kaushal and Kyle Mahowald. 2022. What do tokens know about their characters and how do they know it? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2487–2507, Seattle, United States. Assoc… find
  21. Maxime Kayser, Cornelius Emde, Oana-Maria Camburu, Guy Parsons, Bartlomiej W. Papiez, and Thomas Lukasiewicz. 2022. Explaining chest xray pathologies in natural language. In Medical Image Computing and Computer Assisted Intervention -MICCAI 2022 - 25th International Conference, Singapore, September… find
  22. Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John F. Canny, and Zeynep Akata. 2018. Textual explanations for self-driving vehicles. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science,… find
  23. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4,… find
  24. Joanna Materzynska, Antonio Torralba, and David Bau. 2022. Disentangling visual and written concepts in clip. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16389– 16398. find
  25. J. L. McClelland, D. E. Rumelhart, and PDP Research Group, editors. 1986. Parallel Distributed Processing. Volume 2: Psychological and Biological Models. MIT Press, Cambridge, MA. find
  26. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. find
  27. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  28. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic. find
  29. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. In Advances in Neural Information Processing Systems, volume 33, pages 17153– 17163. Curran Associates, Inc. find
  30. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — In-context Learning and Induction Heads
  31. Barbara H Partee. 1995. Lexical semantics and compositionality. In Lila R. Gleitman and Mark Liberman, editors, Invitation to Cognitive Science, volume 1, pages 311–360. MIT Press, Cambridge, MA. find
  32. Paul Smolensky. 1988. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1):1– 23. find
  33. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR. find
  34. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. find
  35. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations. find
  36. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  37. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). find
  38. Judea Pearl. 2014. Interpretation and identification of causal mediation. Psychological methods, 19. find
  39. Christopher Potts and Roger Levy. 2015. Negotiating lexical uncertainty and speaker expertise with disjunction. In Proceedings of the 41st Annual Meeting of the Berkeley Linguistics Society, pages 417–445, Berkeley, CA. Berkeley Linguistics Society. find
  40. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://cdn.openai.com/ better-language-models/language_models_ are_unsupervised_multitask_learners.pdf. ↗
  41. D. E. Rumelhart, J. L. McClelland, and PDP Research Group, editors. 1986. Parallel Distributed Processing. Volume 1: Foundations. MIT Press, Cambridge, MA. find
  42. Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao. 2023. Explaining black box text modules in natural language with language models. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2023-rigorously-assessing-natural-language-explanations/