Elamur Research⌕
☰
home / papers / 2026 / Multi-component Causal Tracing in Large Language Models

Multi-component Causal Tracing in Large Language Models

Paper · 2026 · Zirui Yan, Dennis Wei, Dmitriy A Katz et al.

Summary

Causal tracing (also called activation patching) finds which internal parts of an LLM drive a given behaviour. Most prior work intervenes on one neuron or one layer at a time. The authors show this misses interactions: on GPT2-small, jointly patching pairs of attention heads or MLP units changes the metric by more than the sum of the individual effects. They therefore frame multi-component causal tracing as choosing the subset of components, such as attention heads or MLP neurons, that together maximise a chosen metric, for example gender bias or factual recall. Exhaustive search over subsets is combinatorially explosive. Their algorithm, PGB-CT, avoids it by replacing hard on/off patches with soft, continuous interventions. A transformed metric makes the problem solvable by gradient descent, and a scheduled penalty pushes the solution towards sparse, binary selections. They test it on GPT2 models, LLaMA and Qwen-sized models across bias and factuality datasets. PGB-CT finds more influential component sets than top-k and random baselines, comes close to greedy search, and runs far faster. This makes multi-component analysis practical for larger models.

Key points

  • On GPT2-medium with WinoGender, PGB-CT runs about 1.76× faster than top-k and 229× faster than greedy, while its scores approach greedy and exceed top-k.
  • On GPT2-small with WinoBias, the heads PGB-CT selects overlap more with greedy's choices (Jaccard 0.64) than with top-k's (0.44).
  • Experiments cover DistilGPT2 (72 heads, 4,608 MLP neurons) through GPT2-xl (1,200 heads, 76,800 neurons). Datasets include WinoBias, WinoGender, Professions, CounterFact and VBD.
  • When heads and individual neurons share one candidate pool, MLP neurons crowd out the heads. GPT2-medium, for instance, has 24,576 neurons against 384 heads. Grouping each layer's neurons into a single block fixes this, and the approach scales to LLaMA-13B (1,400 heads, 40 MLP blocks).
  • Limitations: the method needs a fixed target metric and sparsity-related hyperparameter tuning. Gradient descent offers no global-optimum guarantee, and testing was mostly on English data. Code: github.com/ZiruiYan/multi-component-causal-tracing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Rensselaer Polytechnic InstituteTroy · United States21 works · 4,933 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zirui Yan, Dennis Wei, Dmitriy A Katz, Prasanna Sattigeri and Ali Tajer. 2026. Multi-component Causal Tracing in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.154

@inproceedings{yan2026multi,
  title = {{Multi-component Causal Tracing in Large Language Models}},
  author = {Zirui Yan, Dennis Wei, Dmitriy A Katz, Prasanna Sattigeri and Ali Tajer},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.154},
  url = {https://doi.org/10.18653/v1/2026.acl-long.154}
}

References

56 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. 2024. Fact… — Factuality Challenges in the Era of Large Language Models
  2. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proc. Advances in Neural Information Processing Systems, Barcelona, Spain.
  3. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… find
  4. Yuchen Cai, Ding Cao, Rongxi Guo, Yaqin Wen, Guiquan Liu, and Enhong Chen. 2024. Locating and mitigating gender bias in large language models. In Proc. International Conference on Intelligent Computing, Tianjin, China. find
  5. Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, Xifeng Yan, William Yang Wang, Philip Torr, Dawn Song, and Kai Shu. 2026. Can editing LLMs inject harm? In Proc. AAAI Conference on Artificial Intelligence, Singapore, Si… find
  6. Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. 2020. Generating hierarchical explanations on text classification via feature interaction detection. In Proc. Annual Meeting of the Association for Computational Linguistics, Online. find
  7. Bilal Chughtai, Lawrence Chan, and Neel Nanda. 2023. A toy model of universality: Reverse engineering how networks learn group operations. In Proc. International Conference on Machine Learning, Honolulu, HI. find
  8. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT‘s attention. In Proc. ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy. find
  9. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA. find
  10. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In Proc. International Conference on Learning Representations, Vienna, Austria. find
  11. Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. 2023. Discovering variable binding circuitry with desiderata. In Proc. International Conference on Machine Learning Workshop on Challenges in Deployable Generative AI, Honolulu, HI. find
  12. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. find
  13. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. 2024. A primer on the inner workings of transformer-based language models. Preprint, arXiv:2405.00208. find
  14. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Proc. Advances in Neural Information Processing Systems, Online. find
  15. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proc. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Online.
  16. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. Inducing causal structure for interpretable neural networks. In Proc. International Conference on Machine Learning, Baltimore, MD. find
  17. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proc. Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates.
  18. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proc. Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic. find
  19. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In Proc. Conference on Language Modeling, Philadelphia, PA. find
  20. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA. find
  21. Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology, 33(7):1–30. find
  22. Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. 2022. A survey of algorithmic recourse: contrastive explanations and consequential recommendations. ACM Computing Surveys, 55(5):1–29. find
  23. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA.
  24. AI @ Meta Llama Team. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. ai.meta.com. find
  25. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through L 0 regularization. In Proc. International Conference on Learning Representations, Vancouver, Canada. find
  26. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020. Gender bias in neural natural language processing. In Logic, language, and security: Essays dedicated to Andre Scedrov on the occasion of his 65th birthday, volume 12300, pages 189–202. Springer, Switzerland.
  27. Haoyan Luo and Lucia Specia. 2024. From understanding to utilization: A survey on explainability for large language models. Preprint, arXiv:2401.12874. — [arXiv:2401.12874](https://arxiv.org/abs/2401.12874) find
  28. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA. find
  29. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In Proc. International Conference on Learning Representations, Kigali, Rwanda. find
  30. Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. 2022. GlobEnc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  31. Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. 2023. Quantifying context mixing in transformers. In Proc. Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. find
  32. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 others. 2022. In-context learning and induction heads. Preprint, arXiv:2209.11895.
  33. Lucas Monteiro Paes, Dennis Wei, Hyo Jin Do, Hendrik Strobelt, Ronny Luss, Amit Dhurandhar, Manish Nagireddy, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Werner Geyer, and Soumya Ghosh. 2025. Multi-level explanations for generative language models. In Proc. Annual Meeting of the Association… find
  34. Judea Pearl. 2022. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373–392. Association for Computing Machinery, New York, NY. find
  35. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In Proc. Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, Hon…
  36. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In Proc. International Conference on Learning Representations, Vienna, Austria. find
  37. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI technical report. find
  38. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA.
  39. Victor Sanh, L Debut, J Chaumond, and T Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108. find
  40. Harshay Shah, Andrew Ilyas, and Aleksander Madry. 2024. Decomposing and editing predictions by modeling model computation. In Proc. International Conference on Machine Learning, Vienna, Austria. find
  41. Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. 2024. Adapted large language models can outperform medical experts in…
  42. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. 2020a. Causal mediation analysis for interpreting neural NLP: The case of gender bias. Preprint, arXiv:2004.12265. — [arXiv:2004.12265](https://arxiv.org/abs/2004.12265) find
  43. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020b. Investigating gender bias in language models using causal mediation analysis. In Proc. Advances in Neural Information Processing Systems, Online. find
  44. Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law & Technology, 31:841. find
  45. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In Proc. International Conference on Learning Representations, Kigali, Rwanda. find
  46. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proc. Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing, Online.
  47. Youze Wang, Wenbo Hu, Yinpeng Dong, Jing Liu, Hanwang Zhang, and Richang Hong. 2025. Align is not enough: Multimodal universal jailbreak attack against multimodal large language models. IEEE Transactions on Circuits and Systems for Video Technology. find
  48. Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. 2021. Integrated directional gradients: Feature interaction attribution for neural NLP models. In Proc. Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing, Online… find
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clément Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2019. Huggin…
  50. Aaquib Syed, Can Rager, and Arthur Conmy. 2023. Attribution patching outperforms automated circuit discovery. In Proc. Advances in Neural Information Processing Systems Workshop on Attributing Model Behavior at Scale, New Orleans, LA. find
  51. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. Preprint…
  52. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Y… — The Rise and Potential of Large Language Model Based Agents: A Survey
  53. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  54. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In Proc. International Conference on Machine Learning, Honolulu, HI.
  55. Fred Zhang and Neel Nanda. 2024. Towards best practices of activation patching in language models: Metrics and methods. In Proc. International Conference on Learning Representations, Vienna, Austria. find
  56. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proc. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-multi-component-causal-tracing-large/