Elamur Research⌕
☰
home / papers / 2024 / Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

Paper · 2024 · Michael Hanna, Sandro Pezzelle and Yonatan Belinkov

Summary

Circuit analysis in mechanistic interpretability looks for a small subgraph of a language model that accounts for its behavior on a task. Testing each edge with causal interventions becomes very costly as models grow. GPT-2 small alone has 32,491 edges. The gradient-based shortcut edge attribution patching (EAP) is cheap, and its circuits overlap strongly with manually found ones, but the authors ask whether those circuits are actually faithful. A circuit is faithful if corrupting every edge outside it leaves task behavior intact. They propose EAP-IG, which adapts integrated gradients by averaging gradients along a path between clean and corrupted activations. This addresses EAP's weakness when local gradients are flat. Across tasks such as Capital-Country, Greater-Than, SVA, Hypernymy and Gender-Bias, EAP-IG finds circuits with higher faithfulness than those found by EAP, while activation-patching circuits sometimes still do better. The authors also show that circuit overlap is a poor stand-in for faithfulness. This matters for anyone comparing mechanisms across tasks or models.

Key points

  • EAP scores all edges using two forward passes and one backward pass. EAP-IG keeps this scalability but integrates gradients over intermediate points, giving better edge scores and more faithful circuits.
  • Faithfulness is measured by patching non-circuit edges with activations from corrupted minimal-pair inputs and comparing the task metric with the clean full model.
  • Across task pairs, cross-task faithfulness correlates with node overlap (Pearson 0.86) and edge overlap (0.83). Linear fits reach R² ≈ 0.74.
  • At moderate overlap, overlap predicts little. Hypernymy and Capital-Country share 0.58 node overlap, yet faithfulness is 0.61 in one direction and 0.2 in the other. Two Greater-Than variants with 0.64 overlap are highly faithful in both directions.
  • Code is released at github.com/hannamw/eap-ig. The authors recommend reporting edge scores and testing faithfulness directly rather than relying on overlap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands153 works · 174,584 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel59 works · 7,590 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSoftware · 150 works

Cite

Cite as Michael Hanna, Sandro Pezzelle and Yonatan Belinkov. 2024. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. arXiv:2403.17806.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.17806 · arXiv 2403.17806

@misc{hanna2024have,
  title = {{Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms}},
  author = {Michael Hanna, Sandro Pezzelle and Yonatan Belinkov},
  year = {2024},
  doi = {10.48550/arxiv.2403.17806},
  eprint = {2403.17806},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.17806}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on ”a is b” fail to learn ”b is a”, 2024. URL https://arxiv.org/abs/2309.12288. ↗
  2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: a suite for analyzing large language models across training and… find
  3. Lawrence Chan, Adria Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: A method for rigorously testing interpretability hypotheses. In AI Alignment Forum, pp. 10, 2022. find
  4. Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna, and Oskar van der Wal. Identifying and adapting transformer-components responsible for gender bias in an English language model. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Procee…
  5. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=89ia77nZ8u. ↗
  6. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  7. Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48, 2020. doi: 10.1162/tacl a 00298. URL https://aclanthology.org/2020.tacl-1.3. ↗
  8. Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale, 2024.
  9. Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability illusions in the generalization of simplified models, 2023. URL https://arxiv.org/abs/2312.03656. ↗
  10. Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/ forum?id=RmuXDtjDhG. ↗
  11. Michael Hanna and David Mareček. Analyzing BERT’s knowledge of hypernymy via prompting. In Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neu… find
  12. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=p4PckNQR8k. ↗
  13. Marti A. Hearst. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics, 1992. URL https://aclanthology.org/C92-2082. ↗
  14. János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URL https://arxiv.org/abs/ 2403.00745. ↗
  15. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023. URL https://arxiv.org/abs/2307.09458. ↗
  16. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535, 2016. doi: 10.1162/tacl a 00115. URL https://aclanthology.org/Q16-1037. ↗
  17. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024. URL https://arxiv.org/abs/2403.19647. ↗
  18. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=fpoAYV6Wsk. ↗
  19. Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit faithfulness metrics are not robust, 2024. URL https://arxiv.org/abs/2407.08734. — [arXiv:2407.08734](https://arxiv.org/abs/2407.08734) ↗
  20. Neel Nanda. Attribution Patching: Activation Patching At Industrial Scale, 2023. URL https://www.neelnanda.io/mechanistic-interpretability/attribution-patching. ↗
  21. Neel Nanda and Joseph Bloom. Transformerlens. TransformerLensOrg/TransformerLens, 2022. https://github.com/ ↗
  22. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. Refining targeted syntactic evaluation of language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of t… find
  23. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. URL https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  24. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — In-context Learning and Induction Heads
  25. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=8sKcAWOf2D. ↗
  26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. find
  27. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Iryna Gurevych, Marianna Apidianaki, and Manaal Faruqui (eds.), Proceedings of the Ninth Joint Confere…
  28. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 31…
  29. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi… find
  30. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017. URL https://proceedings.mlr.press/v70/sundararajan17a/sundararajan17a.pdf. ↗
  31. Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery, 2023. URL https://arxiv.org/abs/2310.10348. ↗
  32. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023. URL https://arxiv.org/abs/2310. 15154. ↗
  33. James P Van Overschelde, Katherine A Rawson, and John Dunlosky. Category norms: An updated and expanded version of the battig and montague (1969) norms. Journal of Memory and Language, 50(3):289–335, 2004. ISSN 0749-596X. doi: https://doi.org/ 10.1016/j.jml.2003.10.003. URL https://www.sciencedirect… — [doi:10.1016/j.jml.2003.10.003](https://doi.org/10.1016/j.jml.2003.10.003) find
  34. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processi… find
  35. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern… — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  36. Task IOI Gender-Bias Greater-Than Capital-Country SVA Hypernymy Greater-Than (Price) Greater-Than (Sequence) Country-Capital Metric logit diff logit diff prob diff logit diff prob diff prob diff prob diff prob diff logit diff Clean Baseline 3.80 0.88 0.81 6.12 0.16 0.21 0.65 0.90 6.78 Corrupted Base… find
  37. Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Hf17y6u9BC. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hanna-2024-have-faith-faithfulness-going-circuit/