Elamur Research⌕
☰
home / papers / 2023 / Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

Paper · 2023 · Atticus Geiger, Zhengxuan Wu, Christopher Potts et al.

Summary

Causal abstraction tests whether a neural network faithfully implements a simpler, interpretable algorithm. Earlier versions of this analysis had two limits. They searched over alignments between model variables and neurons by brute force. They also assumed that each high-level variable maps to its own separate group of neurons. The authors propose distributed alignment search (DAS) to remove both limits. DAS learns a rotation of a hidden representation by gradient descent. It then performs interchange interventions on chosen dimensions of the rotated space, so concepts can be spread across many neurons.

The method is tested on two tasks:

  • Hierarchical equality: a small feed-forward network trained on this task fully matches a tree-structured symbolic program.
  • MoNLI: a fine-tuned BERT reaches perfect interchange intervention accuracy (IIA) for a program with negation and lexical-entailment variables.

In both cases, DAS outperforms localist alignments found by brute force. Further analysis shows a difference between the two models. BERT's "entailment" variable breaks down into separate encodings of the two words' identities. The equality network's relations do not break down into representations of the individual inputs. This matters for judging how symbolic computation can emerge inside neural networks.

Key points

  • DAS replaces exhaustive alignment search with a learned change-of-basis (rotation) matrix, trained with interchange intervention training (IIT) objectives. Faithfulness is measured by interchange intervention accuracy (IIA).
  • Hierarchical equality task: the input is (w, x, y, z), and the label is whether (w = x) equals (y = z). DAS finds an alignment with 100% IIA in a feed-forward network.
  • MoNLI experiment: the model is uncased BERT-base (12 layers, hidden size 768, the MultiNLI checkpoint ishan/bert-base-uncased-mnli). Probing the [CLS] token at layer 9 with 256-dimensional interventions gives IIA of 1.00 for the negation + lexical entailment model. The best brute-force localist alignments reach only about 0.47–0.64.
  • At layer 9, the lexical-entailment subspace can be split into identity representations of w_p and w_h with IIA of about 0.97–0.98. This suggests BERT stores a "data structure" of the two words rather than an abstract relation.
  • The paper was published at CLeaR 2024 (PMLR vol. 236) and is available as arXiv:2303.02536.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard and Noah D. Goodman. 2023. Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. arXiv:2303.02536.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.02536 · arXiv 2303.02536

@misc{geiger2023finding,
  title = {{Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations}},
  author = {Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard and Noah D. Goodman},
  year = {2023},
  doi = {10.48550/arxiv.2303.02536},
  eprint = {2303.02536},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.02536}
}

References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior. arXiv:2205.14140, 2022. URL https://arxiv.org/abs/ 2205.14140. ↗
  2. Sander Beckers, Frederick Eberhardt, and Joseph Y. Halpern. Approximate causal abstractions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, 2019. find
  3. Krzysztof Chalupka, Frederick Eberhardt, and Pietro Perona. Causal feature learning: an overview. Behaviormetrika, 44:137–164, 2017. find
  4. Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses, 2022. find
  5. Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. find
  6. OpenReview.net, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD. ↗
  7. Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. arXiv:2112.06837, 2021. URL https://arxiv.org/abs/ 2112.06837. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  9. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. In Proceedings of the 2020 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, November 2020. do…
  10. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics, pages 1–54, 05 2021. ISSN 0891-2017. doi: 10.1162/coli_a_00404. URL https://doi.org/10.1162/coli_a_00404. ↗
  11. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJC… find
  12. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. Posing fair generalization tasks for natural language inference. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
  13. Atticus Geiger, Kyle Richardson, and Chris Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the 2020 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics…
  14. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 9574–9586, 2021. URL https://papers.nips.cc/paper/2021/hash/ 4f5c422f4d49a5a807eda27434231040-Abstract.html. ↗
  15. Atticus Geiger, Alexandra Carstensen, Michael C Frank, and Christopher Potts. Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review, 2022a. doi: 10.1037/rev0000371. URL https://doi.org/10.1037/rev0000371. ↗
  16. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of… find
  17. Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful interpretation of AI models. arXiv:2106.02997, 2023. URL https://arxiv.org/abs/2106.02997. ↗
  18. Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. Inducing character-level structure in subword-based language models with Type-level Interchange Intervention Training. Ms., Stanford University and UT Austin, 2022. URL https://arxiv.org/abs/2212.09897. ↗
  19. Yumi Iwasaki and Herbert A. Simon. Causality and model abstraction. Artificial Intelligence, 67(1): 143–194, 1994. find
  20. Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 202…
  21. Belinda Z. Li, Maxwell I. Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi… find
  22. Zachary C. Lipton. The mythos of model interpretability. Commun. ACM, 61(10):36–43, 2018. doi: 10.1145/3233231. URL https://doi.org/10.1145/3233231. — [doi:10.1145/3233231](https://doi.org/10.1145/3233231) ↗
  23. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. Towards faithful model explanation in NLP: A survey. CoRR, abs/2209.11326, 2022. doi: 10.48550/arXiv.2209.11326. URL https://doi.org/10.48550/arXiv.2209.11326. ↗
  24. Riccardo Massidda, Atticus Geiger, Thomas Icard, and Davide Bacciu. Causal abstraction with soft interventions. In Mihaela van der Schaar, Cheng Zhang, and Dominik Janzing, editors, Conference on Causal Learning and Reasoning, CLeaR 2023, 11-14 April 2023, Amazon Development Center, Tübingen, German… find
  25. PMLR, 2023. URL https://proceedings.mlr.press/v213/ massidda23a.html. ↗
  26. J. L. McClelland, D. E. Rumelhart, and PDP Research Group, editors. Parallel Distributed Processing. Volume 2: Psychological and Biological Models. MIT Press, Cambridge, MA, 1986. find
  27. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf. ↗
  28. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  29. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — In-context Learning and Induction Heads
  30. Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, page 411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001. find
  31. David Premack. The codes of man and beasts. Behavioral and Brain Sciences, 6(1):125–136, 1983. doi: 10.1017/S0140525X00015077. URL https://doi.org/10.1017/S0140525X00015077. — [doi:10.1017/s0140525x00015077](https://doi.org/10.1017/s0140525x00015077) ↗
  32. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Com…
  33. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning (ICML), 2022. URL https://proceedings.mlr.press/v162/ravfogel22a.html. ↗
  34. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Co…
  35. Eigil F. Rischel and Sebastian Weichwald. Compositional abstraction error and a category of causal models. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence (UAI), 2021. find
  36. Paul K. Rubenstein, Sebastian Weichwald, Stephan Bongers, Joris M. Mooij, Dominik Janzing, Moritz Grosse-Wentrup, and Bernhard Schölkopf. Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI), 2017. find
  37. D. E. Rumelhart, J. L. McClelland, and PDP Research Group, editors. Parallel Distributed Processing. Volume 1: Foundations. MIT Press, Cambridge, MA, 1986. find
  38. P. Smolensky. Neural and conceptual interpretation of PDP models. In Parallel Distributed Processing: Explorations in the Microstructure, Vol. 2: Psychological and Biological Models, page 390–431. MIT Press, Cambridge, MA, USA, 1986. ISBN 0262631105. find
  39. Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. MIT Press, 2000. find
  40. Roger K R Thompson, David L Oden, and Sarah T Boysen. Language-naive chimpanzees (pan troglodytes) judge relations between relations in a conceptual matching-to-sample task. Journal of Experimental Psychology: Animal Behavior Processes, 23(1):31—-43, 1997. find
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S…
  42. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias, 2020. find
  43. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv preprint arXiv:2211.00593, 2022. URL https://arxiv.org/abs/2211.00593. ↗
  44. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),…
  45. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
  46. Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal Proxy Models for concept-based model explanations. arXiv:2209.14279, 2022. URL https://arxiv.org/abs/2209.14279. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geiger-2023-finding-alignments-between-interpretable-causal/