Elamur Research⌕
☰
home / papers / 2023 / Localizing Model Behavior with Path Patching

Localizing Model Behavior with Path Patching

Paper · 2023 · Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato et al.

Summary

Interpretability work often claims that a model behavior is concentrated in a small part of a network, but these claims tend to be informal and hard to compare. This Redwood Research paper turns them into testable statements. A hypothesis names the paths through a computational graph that should carry the input–output relationship on a chosen dataset. Every other path receives a counterfactual input instead of the real one, which can be resampled, corrupted, or mean/zero-ablated. A Treeify operation duplicates shared subgraphs so each path gets its own input copy and can be patched on its own. Fidelity is measured by the average unexplained effect (AUE), which is usually reported relative to the average total effect. The method extends the head-to-head path patching of Wang et al. (2022) to arbitrary path sets and relates it to Pearl's path-specific effects. The authors use it to refine an account of induction heads in a small attention-only transformer and to characterize a GPT-2 behavior. They also release open-source tooling. The result is that localization claims become precise, quantitative and refinable step by step.

Key points

  • A hypothesis is formalized as a tuple (G, δ, P, D):
    • G is the computational graph.
    • δ is a dissimilarity measure, such as absolute difference or KL divergence.
    • P is the set of important paths.
    • D is a distribution over reference/counterfactual input pairs.
  • "Proportion explained" is defined as (1 − AUE/ATE)·100%. It can be negative, and it can look misleadingly high when the loss is placed inside the graph.
  • All unimportant paths share one counterfactual input. Appendix A argues this keeps rewrites consistent, handles additive ensembles and mutually cancelling paths, and allows caching.
  • In the attention-only model, head 0.0 acts as a previous-token head, and heads 1.5 and 1.6 show induction patterns. An "uncommon repeat" subset excludes the 200 most frequent validation tokens.
  • The excerpt provided omits the experimental sections, so quantitative results for the induction-head and GPT-2 studies are not included here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato and Aryaman Arora. 2023. Localizing Model Behavior with Path Patching. arXiv:2304.05969.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.05969 · arXiv 2304.05969

@misc{goldowskydill2023localizing,
  title = {{Localizing Model Behavior with Path Patching}},
  author = {Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato and Aryaman Arora},
  year = {2023},
  doi = {10.48550/arxiv.2304.05969},
  eprint = {2304.05969},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.05969}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sander Beckers, Frederick Eberhardt, and Joseph Y Halpern. Approximate causal abstractions. In Uncertainty in Artificial Intelligence, pp. 606–615. PMLR, 2020. find
  2. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
  3. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) find
  4. Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses., 2022. URL https://bit.ly/3WRBhPD. ↗
  5. Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019. find
  6. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023. find
  7. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023. find
  8. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  9. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kern… find
  10. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint arXiv:2106.06087, 2021. find
  11. Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. arXiv preprint arXiv:2004.14623, 2020.
  12. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. find
  13. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of th… find
  14. Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.04709, 2023a. find
  15. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint arXiv:2303.02536, 2023b.
  16. David Lindner, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability, 2023. find
  17. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  18. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020. find
  19. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, T. J. Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario A… — In-context Learning and Induction Heads
  20. Judea Pearl. Direct and indirect effects. CoRR, abs/1301.2300, 2013. URL http://arxiv.org/abs/1301.2300. — [arXiv:1301.2300](https://arxiv.org/abs/1301.2300) ↗
  21. Ofir Press, Noah A Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012.15832, 2020. find
  22. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks, 2023. find
  23. Jérémy Scheurer Scheurer, Phil3, tony, Jacques Thibodeau, and David Lindner. Practical pitfalls of causal scrubbing, 2023. URL https://www.alignmentforum.org/posts/DFarDnQjMnjsKvW8s/practical-pitfalls-of-causal-scrubbing. ↗
  24. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. find
  25. Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016. find
  26. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388–12401, 2020. find
  27. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goldowsky-dill-2023-localizing-model-behavior-path-patching/