Attribution Patching Outperforms Automated Circuit Discovery
Paper · 2023 · Aaquib Syed, Can Rager and Arthur Conmy
Summary
Mechanistic interpretability tries to find circuits, which are small subgraphs of a network that account for a specific behavior. Automated tools such as ACDC rely on activation patching. That method needs an extra forward pass for each edge it tests, so it scales poorly to large models.
The authors propose Edge Attribution Patching (EAP). It replaces exact patching with a first-order Taylor estimate of how the task metric changes when an edge's clean activation is swapped for a corrupted one. Two forward passes and one backward pass yield scores for every edge. Absolute scores are averaged over clean/corrupted prompt pairs, edges are ranked, and the top k are kept.
The authors benchmarked EAP against published circuits for IOI, Greater-Than and Docstring. EAP beat ACDC in ROC AUC on the first two tasks and was nearly tied on Docstring, at far lower cost. However, EAP scores match true patching effects only loosely. Running ACDC on the EAP-pruned graph improves results further, which makes cheap circuit discovery a realistic option for larger models.
Key points
- ROC AUC results:
- IOI: EAP 0.904 vs. ACDC 0.868 (KL divergence) and 0.588 (task metric).
- Greater-Than: EAP 0.896 vs. 0.849 and 0.458.
- Docstring: EAP 0.976 vs. ACDC-KL 0.982, activation patching 0.978 and ACDC with task metric 0.972.
- Cost and setup:
- EAP needs a constant number of passes, whereas ACDC's cost grows rapidly with model size. Full activation patching was too expensive to run on GPT-2 small, which the IOI and Greater-Than tasks use.
- EAP uses task-specific metrics such as logit difference rather than KL divergence, because the KL gradient is zero on the unablated model.
- Weak approximation: Attribution and activation patching scores correlate weakly (R² = 0.27, fit slope 0.531). Concave interpolation curves cause EAP to overestimate some edges.
- Recommended pipeline: Run EAP first, then ACDC on the pruned subgraph; this combination outperformed EAP alone on Docstring. Code is available at github.com/Aaquib111/acdcpp (NeurIPS 2023 ATTRIB Workshop).
- Evaluation caveat: Benchmarking against human-found circuits is a noisy and limited measure of interpretability.
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksSoftware · 119 works
🧭 Topics 1Research subjects and languages the work is about.Interpretability · 149 works
Cite
Cite as Aaquib Syed, Can Rager and Arthur Conmy. 2023. Attribution Patching Outperforms Automated Circuit Discovery. arXiv:2310.10348.
@misc{syed2023attribution,
title = {{Attribution Patching Outperforms Automated Circuit Discovery}},
author = {Aaquib Syed, Can Rager and Arthur Conmy},
year = {2023},
doi = {10.48550/arxiv.2310.10348},
eprint = {2310.10348},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2310.10348}
}References
- Chris Olah. Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases. 2022. URL : — How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model https://www.transformer-circuits.pub/2022/mech-interp-essay. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. “Zoom In: An Introduction to Circuits”.… ↗
- Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal Abstractions of Neural Networks. 2021. URL : https://arxiv.org/abs/2106.02997. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. “Locating and editing factual associations in GPT”. In: Advances in Neural Information P… ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/syed-2023-attribution-patching-outperforms-automated-circuit/