Elamur Research⌕
☰
home / papers / 2024 / Mechanistic Interpretability for AI Safety — A Review

Mechanistic Interpretability for AI Safety — A Review

Paper · 2024 · Leonard Bereska and Efstratios Gavves

Summary

This review gathers the scattered literature on mechanistic interpretability into one structured introduction for AI researchers. Much of that literature has so far existed mainly as blog posts and reading lists. Mechanistic interpretability tries to reverse-engineer what a neural network has learned, ideally down to pseudocode-level algorithms and human-readable concepts. The authors place it among other interpretability approaches as the bottom-up, causal end of the spectrum.

The paper is a conceptual survey rather than an experimental study. It defines features as irreducible units of representation and organizes the central hypotheses about how networks store and compute information. It then reviews methods for causally dissecting model behavior and offers a taxonomy of the field.

On safety, the authors weigh the potential gains in understanding, control and alignment against risks such as capability gains and dual-use. They name scalability, automation and completeness of interpretation as the main open problems. Their recommendations are:

  • sharpen the field's concepts,
  • establish standards,
  • scale the techniques up,
  • extend the work beyond language models to vision and reinforcement learning.

Key points

  • The paper distinguishes four interpretability paradigms: behavioral (input-output relations), attributional (gradient-based tools such as SmoothGrad, Integrated Gradients and GradCAM), concept-based (probes and representation engineering) and mechanistic.
  • A feature is defined as a basic representational unit that cannot be split into simpler independent factors. This generalizes the irreducibility notion of Engels et al. (2024) to abstract concepts.
  • The organizing hypotheses are linear representation, superposition, universality of circuits and motifs, simulation and prediction orthogonality. Polysemantic neurons are treated as a key obstacle.
  • The paper has eight sections, running from paradigms through methods, taxonomy and AI safety relevance to challenges and future directions. The authors position it as a companion to two reviews written around the same time, one by Rai et al. (2024) and the other by Ferrando et al. (2024).
  • The authors are at the University of Amsterdam (arXiv v3, August 2024), and an HTML version is available on the first author's blog.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands151 works · 174,506 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Bias, fairness & safety · 587 worksInterpretability · 149 works

Cite

Cite as Leonard Bereska and Efstratios Gavves. 2024. Mechanistic Interpretability for AI Safety — A Review. arXiv:2404.14082.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.14082 · arXiv 2404.14082

@misc{bereska2024mechanistic,
  title = {{Mechanistic Interpretability for AI Safety — A Review}},
  author = {Leonard Bereska and Efstratios Gavves},
  year = {2024},
  doi = {10.48550/arxiv.2404.14082},
  eprint = {2404.14082},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.14082}
}

References

374 references, 68 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. CVPR, June 2022. 30 find
  2. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. ICLR, 2016. 9, 14 find
  3. Uri Alon. An introduction to systems biology: design principles of biological circuits. Chapman and Hall/CRC, 2019. 21 find
  4. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. CoRR, 2024. 9
  5. Aryaman Arora, Dan Jurafsky, and Christopher Potts. Causalgym: Benchmarking causal interpretability methods on linguistic tasks. CoRR, February 2024. 19
  6. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. TACL, December 2018. 5, 34
  7. Leopold Aschenbrenner. Situational awareness: The decade ahead. Series: Situational Awareness. June, 2024. 31 find
  8. Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, July 2015. 2 find
  9. Nicholas Bai, Rahul Ajay Iyer, Tuomas Oikarinen, and Tsui-Wei Weng. Describe-and-dissect: Interpreting neurons in vision networks with language models. ICML MI Workshop, June 2024. 23 find
  10. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  11. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. CoRR, June 2021. 3 find
  12. Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. NeurIPS, 2022. 23 find
  13. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. ICLR, December 2018. 29 find
  14. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. CoRR, September 2021. 3, 9, 14, 19
  15. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. CoRR, August 2023. 13, 15
  16. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? ACM FAccT, March 2021. 11 find
  17. Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. CoRR, February 2019. 28 find
  18. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, St… find
  19. Leonard Bereska. Mechanistic interpretability for adversarial robustness — a proposal. Leonard Bereska’s Blog, August 2024. 28 find
  20. Leonard Bereska and Efstratios Gavves. Taming simulators: Challenges, pathways and vision for the alignment of large language models. AAAI-SS, October 2023. 24, 30, 34 find
  21. Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. CoRR, December 2019. 28 find
  22. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. OpenAI Blog, 2023. 5, 23, 25 find
  23. Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. Proc. Natl. Acad. Sci. U.S.A., January 2024. 2 find
  24. Christopher M. Bishop. Pattern recognition and machine learning. Springer-Verlag New York Inc., 2006. 4 find
  25. Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens. CoRR, November 2022. 9, 28 find
  26. Joseph Bloom and Jay Bailey. Features and adversaries in memorydt. LessWrong, October 2023. 29, 30 find
  27. Joseph Bloom and Paul Colognese. Decision transformer interpretability. AI Alignment Forum, 2023. 30 find
  28. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Vi’egas, and M. Wattenberg. An interpretability illusion for bert. CoRR, April 2021. 29 find
  29. Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. ICML MI Workshop, May 2024. 19 find
  30. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tr… find
  31. Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. CoRR, February 2024. 22 find
  32. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. CoRR, April 2023.…
  33. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. ICLR, 2023. 3, 13, 14, 24, 29 find
  34. Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill, Dan Braun, Jake Mendel, Kaarel Hanni, Avery Griffin, Jorn Stohler, Magdalena Wache, and Marius Hobbhahn. The local interaction basis: Identifying computationally-relevant and sparsely interacting features in neural networks. CoRR, May 2024… find
  35. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. ICLR, October 2022. 21 find
  36. Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, June 2020. 10, 15, 21, 30, 32 find
  37. Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. 10, 21, 30, 32 find
  38. Steven Cao, Victor Sanh, and Alexander M. Rush. Low-complexity probing via finding subnetworks. NAACL-HLT, April 2021. 14 find
  39. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, March 2019. 28 find
  40. Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. ECML PKDD, 2018. 2 find
  41. Stephen Casper. The engineer’s interpretability sequence. AI Alignment Forum, February 2023. 23, 25, 27, 28, 29 find
  42. Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. NeurIPS, October 2021. 28 find
  43. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar… find
  44. Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield-Menell. Diagnostics for deep neural networks with automated copy/paste attacks. NeurIPS 2022 ML Safety Workshop (Best paper award), May 2023b. 25 find
  45. Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red teaming deep neural networks with feature synthesis tools. NeurIPS, 2023c. 25, 29 find
  46. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Kru… find
  47. Lawrence Chan. What i would do if i wasn’t at arc evals. AI Alignment Forum, May 2023. 24, 28, 29 find
  48. Lawrence Chan, Adrià Garriga-alonso, Nicholas Goldowsky-Dill, ryan_greenblatt, jenny, Ansh Radhakrishnan, Buck, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses [redwood research]. AI Alignment Forum, December 2022. 13, 19 find
  49. Lawrence Chan, Leon Lang, and Erik Jenner. Natural abstractions: Key claims, theorems, and critiques. AI Alignment Forum, March 2023. 4, 10, 33 find
  50. David Chanin, Anthony Hunter, and Oana-Maria Camburu. Identifying linear relational concepts in large language models. CoRR, 2023. 9 find
  51. Charbel-Raphaël. Against almost every theory of impact of interpretability. AI Alignment Forum, August 2023. 25 find
  52. Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. CoRR, November 2023a. 11 find
  53. Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian phase transitions in a toy model of superposition. CoRR, October 2023b. 21, 25 find
  54. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, December 2017. 30 find
  55. Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge, January 2021. 24, 32 find
  56. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. ICML, 2023. 11, 22, 25 find
  57. Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms. NeurIPS Workshop Attributing Model Behaviour at Scale, 2024. 10 find
  58. Paul Colognese. Internal target information for ai oversight. LessWrong, 2023. 30 find
  59. Paul Colognese and Jozdien. High-level interpretability: detecting an ai’s objectives. AI Alignment Forum, 2023. 30 find
  60. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. NeurIPS, 2023. 18, 19, 22, 25, 30 find
  61. Ian C. Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: a unified framework for model explanation. J. Mach. Learn. Res., January 2021. 2 find
  62. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. ICLR, January 2024. 8, 9, 13, 15, 16, 19, 29 find
  63. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. ACL, 2022. 5, 29 find
  64. David "davidad" Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, and Joshua Tenenbaum. Towards guaranteed safe ai:… find
  65. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. Proceedings of the AAAI Conference on Artificial Intelligence, July 2019. 14 find
  66. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. ACL, December 2022. 15
  67. Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata. CoRR, July 2023. 10, 22 find
  68. Mingyang Deng, Lucas Tao, and Joe Benton. Measuring feature sparsity in language models. CoRR, 2023. 9, 15 find
  69. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language models.… find
  70. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. CoRR, March 2023. 15 find
  71. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. CoRR, March 2017. 21, 29, 30
  72. Keke Du, Shan Chang, Huixiang Wen, and Hao Zhang. Fighting adversarial images with interpretable gradients. ACM TURC, October 2021. 28 find
  73. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. ICML MI Workshop, June 2024. 16 find
  74. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. EMNLP, October 2020. 5 find
  75. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. TACL, February 2021. 14
  76. Nelson Elhage, Tristan Hume, Olsson Catherine, Nanda Neel, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Con… find
  77. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. Transformer Circuits Thread, 2022b. 5, 6, 7, 9, 15, 25, 26, 27, 28, 29, 30, 34, 35 find
  78. Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. CoRR, May 2024. 4, 8, 33 find
  79. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. CoRR, September 2019. 28 find
  80. Ingrid Lossius Falkum and Agustin Vicente. Polysemy: Current perspectives and approaches. Lingua, April 2015. 34 find
  81. Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. Challenges with unsupervised llm knowledge discovery. CoRR, 2023. 14 find
  82. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, May 2021. 18
  83. Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? CoRR, October 2023. 10, 22, 26, 30 find
  84. Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. CoRR, February 2024. 18, 22
  85. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of transformer-based language models. CoRR, May 2024. 2
  86. Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. CoRR, May 2023. 23, 30 find
  87. Stanislav Fort and Balaji Lakshminarayanan. Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness. CoRR, August 2024. 28 find
  88. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, March 2019. 20 find
  89. Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability illusions in the generalization of simplified models. CoRR, 2023a. 29, 30 find
  90. Dan Friedman, Alexander Wettig, and Danqi Chen. Learning transformer programs. NeurIPS, June 2023b. 20 find
  91. Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. ICLR, 2023a. 25 find
  92. Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. CoRR, October 2023b. 26 find
  93. Zach Furman and Edmund Lau. Estimating the local learning coefficient at scale. CoRR, February 2024. 21 find
  94. Jorge García-Carrasco, Alejandro Maté, and Juan Trujillo. Detecting and understanding vulnerabilities in language models via mechanistic interpretability. IJCAI, August 2024. 28, 29 find
  95. Albert Garde, Esben Kran, and Fazl Barez. Deepdecipher: Accessing and investigating neuron activation in large language models. NeurIPS Workshop XAIA, October 2023. 13 find
  96. Peter Gardenfors. Conceptual spaces: The geometry of thought. MIT press, 2004. 12 find
  97. Charles J. Garfinkle and Christopher J. Hillar. On the uniqueness and stability of dictionaries for sparse representation of noisy signals. IEEE Transactions on Signal Processing, December 2019. 15 find
  98. Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs. CoRR, May 2024. 19 find
  99. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. NeurIPS, 2021a. 18 find
  100. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. ICML, January 2021b. 17 find
  101. Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. CoRR, January 2023a. 13, 16, 18, 19 find
  102. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. CoRR, 2023b. 13, 18
  103. Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. CoRR, March 2019. 20 find
  104. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. EMNLP, October 2022. 15
  105. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. EMNLP, October 2023. 17 find
  106. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. CoRR, January 2024. 18
  107. Amirata Ghorbani and James Zou. Neuron shapley: Discovering the responsible neurons. NeurIPS, November 2020. 5, 29 find
  108. Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, March 2021. 5, 30 find
  109. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. CoRR, 2023. 13, 17, 18, 29, 30
  110. Liv Gorton. The missing curve detectors of inceptionv1: Applying sparse autoencoders to inceptionv1 early vision. ICML MI Workshop, June 2024. 15 find
  111. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. CoRR, 2023. 29 find
  112. Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. CoRR, November 2020. 28 find
  113. Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. ICML MI Workshop, June 2024. 29 find
  114. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalizatio… — Studying Large Language Model Generalization with Influence Functions
  115. Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. ICML MI Workshop, June 2024. 25 find
  116. Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. Interpbench: Semi-synthetic transformers for evaluating mechanistic interpretability techniques. CoRR, July 2024. 29 find
  117. Wes Gurnee and Max Tegmark. Language models represent space and time. ICLR, 2024. 12 find
  118. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. TMLR, 2023. 6, 13, 14
  119. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. CoRR, January 2024. 11 find
  120. David R. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. NeurIPS, September 2018. 11 find
  121. Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let’s agree to agree: Neural networks share classification order on real datasets. ICML, 2020. 11 find
  122. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023. 10, 17, 22, 30
  123. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. ICML MI Workshop, June 2024. 29 find
  124. Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. ICML MI Workshop, August 2024. 29 find
  125. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. NeurIPS Spotlight, January 2023. 29 find
  126. Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, 2024. 15 find
  127. Stefan Heimersheim and Jett. A circuit for python docstrings in a 4-layer attention-only transformer. AI Alignment Forum, February 2023. 22 find
  128. Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. EMNLP, October 2023. 9, 17, 22
  129. Dan Hendrycks. Introduction to AI Safety, Ethics, and Society. Self-published, 2023a. 26 find
  130. Dan Hendrycks. Natural selection favors ais over humans. CoRR, July 2023b. 24 find
  131. Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. CoRR, June 2022. 1, 24, 25 find
  132. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. CoRR, June 2022. 26
  133. Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. CoRR, October 2023. 1, 24 find
  134. Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. 6, 27 find
  135. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretabilit… — Scaling Laws and Interpretability of Learning from Repeated Data
  136. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. CoRR, August 2023. 9
  137. John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. NAACL HLT, June 2019. 14
  138. Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, December 2018. 27 find
  139. Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. 30 find
  140. Geoffrey E Hinton. Distributed representations. Carnegie Mellon University, 1984. 27 find
  141. Marius Hobbhahn. Marius’ alignment agenda, 2022. 30 find
  142. Marius Hobbhahn and Lawrence Chan. Should we publish mechanistic interpretability research? AI Alignment Forum, April 2023. 25 find
  143. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  144. Jesse Hoogland, Liam Carroll, and Daniel Murfet. Stagewise development in neural networks. AI Alignment Forum, March 2024. 21 find
  145. Jing Huang, Atticus Geiger, Karel D’Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously assessing natural language explanations of neurons. CoRR, September 2023. 5
  146. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluating interpretability methods on disentangling language model representations. CoRR, 2024. 19 find
  147. Evan Hubinger. Chris olah’s views on agi safety. AI Alignment Forum, November 2019a. 4, 24, 26 find
  148. Evan Hubinger. Gradient hacking. AI Alignment Forum, October 2019b. 26 find
  149. Evan Hubinger. An overview of 11 proposals for building safe advanced ai. CoRR, December 2020. 24 find
  150. Evan Hubinger. A transparency and interpretability tech tree. AI Alignment Forum, June 2022. 9, 24, 26, 30 find
  151. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. CoRR, May 2019. 24, 31 find
  152. Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Conditioning predictive models: Risks and strategies. CoRR, February 2023. 24, 31 find
  153. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Selli… — Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  154. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurIPS, August 2019. 4, 28, 29 find
  155. M. Ivanitskiy, Alexander F. Spies, Tilman Rauker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Structured world representations in maze-solving transformers. CoRR, December 2023. 12, 30, 31 find
  156. Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? CoRR, April 2020. 29
  157. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. CoRR, November 2023. 30 find
  158. Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study. ICML MI Workshop, June 2024. 30 janus. find
  159. Simulators. LessWrong, September 2022. 11, 12, 24, 34 find
  160. Erik Jenner, Adrià Garriga-alonso, and Egor Zverev. A comparison of causal scrubbing, causal abstractions, and related methods. AI Alignment Forum, June 2023. 13, 18, 19 find
  161. Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network. CoRR, June 2024. 24 find
  162. Adam Jermyn, Chris Olah, and T Henighan. Circuits updates - may 2023: Attention head superposition. find
  163. Transformer Circuits Thread, 2023. 29 find
  164. Adam S. Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. CoRR, November 2022. 6, 20, 25 find
  165. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo,… — AI Alignment: A Comprehensive Survey
  166. Jozdien. Conditioning generative models for alignment. AI Alignment Forum, July 2022. 24 find
  167. Jaap Jumelet. Evaluating and interpreting language models. NLP Lecture, November 2023. 2 find
  168. Amlan Jyoti, Karthik Balaji Ganesh, Manoj Gayala, Nandita Lakshmi Tunuguntla, Sandesh Kamath, and Vineeth N. Balasubramanian. On the robustness of explanations of deep neural network models: A survey. CoRR, November 2022. 28 find
  169. Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. COLM, July 2024. 11 find
  170. Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, C. M. Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretability with board game models. ICML MI Workshop (Oral), July 2024. 15 find
  171. Theodoros Kasioumis, Joe Townsend, and Hiroya Inakoshi. Elite backprop: Training sparse interpretable neurons. International Workshop on Neuro-Symbolic Learning and Reasoning, 2021. 20 find
  172. Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. CoRR, July 2021. 28 find
  173. Jan Kirchner. Neuroscience and natural abstractions. LessWrong, March 2023. 11 find
  174. Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. ICML MI Workshop, June 2024. 15 find
  175. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, July 2019. 3, 10
  176. János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components. CoRR, March 2024. 22 find
  177. Nicholas Kross. Why and when interpretability work is dangerous. LessWrong, May 2023. 25 find
  178. Jan Kulveit. Risks from ai misalignment at different scales, July 2024. 24 find
  179. Jan Kulveit, Clem von Stengel, and Roman Leventov. Predictive minds: Llms as atypical active inference agents. CoRR, November 2023. 11 find
  180. Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. CoRR, 2023. 29 find
  181. Michael Lan and Fazl Barez. Locating cross-task sequence continuation circuits in transformers. CoRR, November 2023. 10 find
  182. Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges with partial observability of human evaluators in reward learning. CoRR, March 2024. 28, 32 find
  183. Georg Lange, Alex Makelov, and Neel Nanda. An interpretability illusion for activation patching of arbitrary subspaces. AI Alignment Forum, August 2023. 18 find
  184. Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet. Decoderlens: Layer-wise interpretation of encoder-decoder transformers. CoRR, 2023. 15 find
  185. Edmund Lau, Daniel Murfet, and Susan Wei. Quantifying degeneracy in singular models via the learning coefficient. CoRR, August 2023. 21 find
  186. Connor Leahy. Barriers to mechanistic interpretability for agi safety. AI Alignment Forum, 2023. 26 find
  187. Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability research. CoRR, October 2020. 29 find
  188. Victor Lecomte, Kushal Thaman, Trevor Chow, Rylan Schaeffer, and Sanmi Koyejo. Incidental polysemanticity. CoRR, 2023. 27, 28, 29 find
  189. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. CoRR, 2024. 30
  190. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. ACL-IJCNLP, August 2021. 12 find
  191. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023a. 8, 11 find
  192. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. NeurIPS Spotlight, July 2023b. 9
  193. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? NIPS Workshop on Feature Extraction, December 2015. 11 find
  194. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. CoRR, July 2023. 10, 17, 22, 29, 30
  195. David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. CoRR, 2023. 29 find
  196. Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does roberta know and when? EMNLP, September 2021. 30
  197. Ziming Liu and Max Tegmark. A neural scaling law from lottery ticket ensembling. CoRR, 2023. 21 find
  198. Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. NeurIPS, 2022a. 21, 32 find
  199. Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. ICML, 2022b. 21 find
  200. Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, June 2023a. 13, 20, 23, 29 find
  201. Ziming Liu, Mikail Khona, Ila R. Fiete, and Max Tegmark. Growing brains: Co-emergence of anatomical and functional modularity in recurrent neural networks. CoRR, 2023b. 20 find
  202. Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as compression: A nonlinear complexity perspective. CoRR, 2023c. 21 find
  203. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. CoRR, June 2019. 27 find
  204. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. CoRR, August 2024. 26
  205. Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models. CoRR, 2024. 10 find
  206. Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M. Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. NeurIPS, October 2018. 28 find
  207. Aleksandar Makelov. Sparse autoencoders match supervised features for model steering on the ioi task. ICML MI Workshop, June 2024. 15 find
  208. Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. CoRR, May 2024. 15 find
  209. Alex Mallen and Nora Belrose. Eliciting latent knowledge from quirky language models. CoRR, December 2023. 32 find
  210. Narek Maloyan, Ekansh Verma, Bulat Nutfullin, and Bislan Ashinov. Trojan detection in large language models: Insights from the trojan detection challenge. CoRR, April 2024. 28 find
  211. Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning: Universal fourier features emerge in invariant networks. CoRR, December 2023. 11 find
  212. Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, and Fazl Barez. Interpreting reward models in rlhf-tuned language models using sparse autoencoders. CoRR, October 2023a. 15 find
  213. Luke Marks, Amir Abdullah, Clement Neo, Rauno Arike, Philip Torr, and Fazl Barez. Beyond training objectives: Interpreting reward model divergence in large language models. CoRR, October 2023b. 31 find
  214. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, March 2024. 19 find
  215. Simon C. Marshall and Jan H. Kirchner. Understanding polysemanticity in neural networks through coding theory. CoRR, January 2024. 6, 27, 28 find
  216. Mantas Mazeika, Andy Zou, Akul Arora, Pavel Pleskov, Dawn Song, Dan Hendrycks, Bo Li, and David Forsyth. How hard is trojan detection in dnns? fooling detectors with evasive trojans. CoRR, September 2022. 29 find
  217. Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. CoRR, October 2023. 10 find
  218. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. PNAS, November 2022. 13, 14, 26 find
  219. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. CoRR, July 2023. 18, 26, 28, 32 find
  220. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NeurIPS, 2022a. 13, 17, 24, 29 find
  221. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. ICLR, 2022b. 29 find
  222. William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. CoRR, 2023. 21 find
  223. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. A mechanism for solving relational tasks in transformer language models. CoRR, May 2023. 30 find
  224. Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. CoRR, March 2023. 9, 21, 30
  225. Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. CoRR, February 2024. 23, 27 find
  226. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. NeurIPS, October 2013. 9
  227. Joseph Miller and Clement Neo. We found an neuron in gpt-2. AI Alignment Forum, February 2023. 22 find
  228. Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, February 2019. 29
  229. Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt Turner. Understanding and controlling a maze-solving policy network. CoRR, October 2023. 30 find
  230. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kıcıman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia. CoRR, 2023. 30 find
  231. Basel Mousi, Nadir Durrani, and Fahim Dalvi. Can llms facilitate interpretation of pre-trained language models? EMNLP, 2023. 23 find
  232. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. NeurIPS, June 2020. 5, 25, 29 find
  233. Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretabi… find
  234. Jatin Nainani. Evaluating brain-inspired modular training in automated circuit discovery for mechanistic interpretability. CoRR, January 2024. 22, 23 find
  235. Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. NeurIPS, May 2019. 21 find
  236. Neel Nanda. 200 cop in mi: Looking for circuits in the wild. Neel Nanda’s Blog, 2022a. 30 find
  237. Neel Nanda. 200 cop in mi: Analysing training dynamics. Neel Nanda’s Blog, 2022b. 30 find
  238. Neel Nanda. 200 cop in mi: Studying learned features in language models. Neel Nanda’s Blog, 2022c. 30 find
  239. Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary. Neel Nanda’s Blog, December 2022d. 2 find
  240. Neel Nanda. A longlist of theories of impact for interpretability. AI Alignment Forum, March 2022e. 23 find
  241. Neel Nanda. 200 cop in mi: Exploring polysemanticity and superposition. Neel Nanda’s Blog, January 2023a. 29 find
  242. Neel Nanda. 200 cop in mi: Techniques, tooling and automation. Neel Nanda’s Blog, 2023b. 30 find
  243. Neel Nanda. Actually, othello-gpt has a linear emergent world representation. Neel Nanda’s Blog, March 2023c. 8 find
  244. Neel Nanda. Attribution patching: Activation patching at industrial scale. Neel Nanda’s Blog, February 2023d. 18 find
  245. Neel Nanda. How to think about activation patching. AI Alignment Forum, April 2023e. 18, 21 find
  246. Neel Nanda. Mechanistic interpretability quickstart guide. Neel Nanda’s Blog, January 2023f. 2, 30 find
  247. Neel Nanda. An extremely opinionated annotated list of my favourite mechanistic interpretability papers v2. AI Alignment Forum, July 2024. 2 find
  248. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR, January 2023a. 9, 21, 22, 23, 25, 29, 30
  249. Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, September 2023b. 8, 11, 28
  250. Neel Nanda, S. Rajamanoharan, J. Kramár, and R. Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. AI Alignment Forum, 2023c. URL https://www.alignmentforum. org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall, 2023c. 10 ↗
  251. Geraldin Nanfack, Alexander Fulleringer, Jonathan Marty, Michael Eickenberg, and Eugene Belilovsky. Adversarial attacks on the interpretation of neuron activation maximization. Proceedings of the AAAI Conference on Artificial Intelligence, March 2024. 13 find
  252. Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. CoRR, December 2022. 24
  253. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. CoRR, October 2022. 24, 25 find
  254. Eshaan Nichani, Alex Damian, and Jason D. Lee. How transformers learn causal structure with gradient descent. CoRR, February 2024. 11 find
  255. NicholasKees and janus. Searching for search. AI Alignment Forum, November 2022. 24 nostalgebraist. interpreting gpt: the logit lens. find
  256. AI Alignment Forum, August 2020. 13, 14, 19 find
  257. Chris Olah. Distributed representations: Composition & superposition. Transformer Circuits Thread, 2023. 27 find
  258. Chris Olah and Shan Carter. Research debt. Distill, March 2017. 2 find
  259. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, November 2017. 29 find
  260. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, March 2018. 2 find
  261. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, March 2020. 2, 4, 5, 9, 10, 21, 32, 34 find
  262. Christopher Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. Transformer Circuits Thread, 2022. 2, 4 find
  263. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: a strategy employed by v1? Vision Res, December 1997. 15, 16 find
  264. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — In-context Learning and Induction Heads
  265. Laura O’Mahony, Vincent Andrearczyk, Henning Muller, and Mara Graziani. Disentangling neuron representations with concept vectors. CVPR Workshops, April 2023. 9 find
  266. Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. CoRR, May 2024. 15, 19 find
  267. Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. ACL 2024, June 2024. 10
  268. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  269. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. CoNLL, 2023. 15, 29
  270. Vedant Palit, Rohan Pandey, Aryaman Arora, and P. Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for blip. ICCVW, 2023. 30 find
  271. Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vision transformers. ICML MI Workshop, June 2024. 30 find
  272. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. NeurIPS Workshop on Causal Representation Learning, November 2023a. 28
  273. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. ICML MI Workshop (Oral), June 2024. 9 find
  274. Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. CoRR, August 2023b. 24 find
  275. Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. ICLR, 2022. 12 find
  276. Judea Pearl. Causality. Cambridge University Press, 2009. 11, 16 find
  277. Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Penguin Books Limited, May 2018. 28 find
  278. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, November 2015. 28 find
  279. Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, October 2017. 28 find
  280. Nicky Pochinkov and Nandi . Machine unlearning evaluations as interpretability benchmarks. AI Alignment Forum, August 2023. 25 find
  281. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. ICML, April 2023. 25 find
  282. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, January 2022. 21
  283. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. ICLR, February 2024. 30 find
  284. Philip Quirke and Fazl Barez. Understanding addition in transformers. CoRR, October 2023. 9, 29, 30 find
  285. Philip Quirke, Clement Neo, and Fazl Barez. Increasing trust in language models through the reuse of verified circuits. CoRR, February 2024. 29 find
  286. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. CoRR, July 2024. 2 find
  287. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. CoRR, April 2024. 16
  288. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. TMLR, August 2023. 1, 13, 25, 26, 27, 28, 29 find
  289. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. ACL, July 2020. 18
  290. Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence of sparse concepts in dnns. CoRR, April 2023. 27 find
  291. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. NAACL, August 2016. 2, 18
  292. RicG. Agi-automated interpretability is suicide. LessWrong, May 2023. 25, 30 find
  293. Jonathan Richens and Tom Everitt. Robust agents learn causal world models. ICLR Oral, February 2024. 11 find
  294. Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh Srivastava. Logical neural networks. NeurIPS, June 2020. 20, 27 find
  295. Mateo Rojas-Carulla, Bernhard Scholkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018. 28 find
  296. Andrew Slavin Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. AAAI, November 2017. 28 find
  297. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell, May 2019. 29
  298. Cody Rushing and Neel Nanda. Explorations of self-repair in language models. ICML, May 2024. 18 find
  299. Thane Ruthenis. Internal interfaces are a high-priority interpretability target. AI Alignment Forum, December 2022. 24 find
  300. Thane Ruthenis. World-model interpretability is all we need. AI Alignment Forum, January 2023. 24 find
  301. Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. TACL, November 2022. 5
  302. Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models. CoRR, September 2023a. 9 find
  303. Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. CoRR, October 2023b. 15 find
  304. Emmanuelle Salin, Badreddine Farah, Stéphane Ayache, and Benoit Favre. Are vision-language transformers learning multimodal representations? a probing perspective. AAAI, June 2022. 30 find
  305. Tommaso Salvatori, Ankur Mali, Christopher L. Buckley, Thomas Lukasiewicz, Rajesh P. N. Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. CoRR, 2023. 11 find
  306. Naomi Saphra. Interpretability creationism. The Gradient, 2023. 21 find
  307. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? CoRR, May 2023. 20
  308. Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. CoRR, July 2023. 6, 26, 27 find
  309. Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. Special Issue of Proceedings of the IEEE - Advances in Machine Learning and Deep Neural Networks, February 2021. 27 find
  310. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. NeurIPS Oral, October 2022. 25 find
  311. Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. ICCV, 2016. 2 find
  312. Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, November 2023. 11, 12, 24
  313. Lloyd S. Shapley. A value for n -person games. Cambridge University Press, October 1988. 2 find
  314. Lee Sharkey. Circumventing interpretability: How to defeat mind-readers. CoRR, December 2022. 24, 26 find
  315. Lee Sharkey. A technical note on bilinear layers for interpretability. CoRR, May 2023. 20 find
  316. Lee Sharkey, Sid Black, and beren. Current themes in mechanistic interpretability research. AI Alignment Forum, November 2022a. 2, 8 find
  317. Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders. AI Alignment Forum, 2022b. 8, 15, 16 find
  318. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards… — Towards Understanding Sycophancy in Language Models
  319. Justin Shovelain and Elliot McKernon. The risk-reward tradeoff of interpretability research. LessWrong, July 2023. 25 find
  320. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, 2017. 2
  321. James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. ICML, May 2023. 20 slavachalnev. Sparse mlp distillation. find
  322. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, June 2017. 2 find
  323. Nate Soares. If interpretability research goes well, it may get dangerous. LessWrong, April 2023. 25 find
  324. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014. 5
  325. Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. CoRR, 2023. 21 find
  326. Jacob Steinhardt. Emergent deception and emergent optimization. Bounded Regret, February 2023. 20, 23, 28 find
  327. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. EMNLP, October 2023. 17, 22 find
  328. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Rob… find
  329. Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov, Max Vladymyrov, and Mark Sandler. Learning and unlearning of fabricated knowledge in language models. ICML MI Workshop, June 2024. 25 find
  330. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML, June 2017. 2 find
  331. Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. CoRR, October 2023. 18, 19, 22 technicalities and Stag. Shallow review of live agendas in alignment & safety.
  332. Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi. CoRR, September 2023. 24, 29, 35 find
  333. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, and Brian Chen. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. 15 find
  334. Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. ACL, August 2019. 14
  335. Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. CoRR, 2022. 21 find
  336. Hannes Thurnherr and Jérémy Scheurer. Tracrbench: Generating interpretability testbeds with large language models. ICML MI Workshop, June 2024. 29 find
  337. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. ICML MI Workshop, June 2024. 9 find
  338. Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR, 2023. 9
  339. Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim. Learning to synthesize programs as interpretable and generalizable policies. NeurIPS, 2021. 27 find
  340. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. CoRR, September 2023. 9 find
  341. Dmitry Vaintrob, jake_mendel, and Kaarel. Toward a mathematical framework for computation in superposition. AI Alignment Forum, 2024. 29 find
  342. Alexandre Variengien and Eric Winsor. Look before you leap: A universal emergent decomposition of retrieval tasks in language models. ICML MI Workshop, December 2023. 11, 22, 30 find
  343. Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. CoRR, September 2023. 21, 22 find
  344. Abhinav Verma, Hoang M. Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. NeurIPS, 2019a. 27 find
  345. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. CoRR, April 2019b. 27 find
  346. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. NeurIPS, 2020. 17, 18 find
  347. M. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. CoRR, 2023. 30 find
  348. Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. EMNLP, March 2020. 14
  349. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, ngram, positional. CoRR, September 2023. 5 find
  350. Julius von Kügelgen, M. Loog, A. Mey, and B. Scholkopf. Semi-supervised learning, causality, and the conditional cluster assumption. Conference on Uncertainty in Artificial Intelligence, May 2019. 28 find
  351. Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers. CoRR, September 2023. 24, 26 find
  352. Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. Branch specialization. Distill, April 2021. 10, 21 find
  353. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. ICML MI Workshop, June 2024. 21 find
  354. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. ICLR, 2023. 10, 13, 17, 18, 19, 22, 30, 32 find
  355. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 2020. 2
  356. Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009. 21 find
  357. Sumio Watanabe. Mathematical Theory of Bayesian Statistics. Chapman and Hall, 1 edition, April 2018. 21 find
  358. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, October 2022. 20,…
  359. John Wentworth. How to go from interpretability to alignment: Just retarget the search. AI Alignment Forum, August 2022. 24 find
  360. James C. R. Whittington, Will Dorrell, Surya Ganguli, and Timothy E. J. Behrens. Disentangling with biological constraints: A theory of functional cell types. CoRR, September 2022. 15 find
  361. Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Back-doorbench: A comprehensive benchmark of backdoor learning. NeurIPS Datasets and Benchmarks, October 2022a. 29 find
  362. Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. Causal distillation for language models. NAACL-HLT, July 2022b. 19 find
  363. Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal proxy models for concept-based model explanations. ICML, 2023a. 13, 19 find
  364. Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. CoRR, May 2023b. 18, 22
  365. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models. CoRR, May 2024. 19
  366. Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. CoRR, October 2023. 10 find
  367. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. ECCV, 2014. 13
  368. Biao Zhang, Ivan Titov, and Rico Sennrich. Sparse attention with linear units. EMNLP, October 2021. 20 find
  369. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, February 2017. 21 find
  370. Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. ICLR, 2023. 18
  371. Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees. CVPR, 2019. 27 find
  372. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. CoRR, 2023. 22 find
  373. Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activations? NeurIPS, November 2021. 13 find
  374. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendry… — Representation Engineering: A Top-Down Approach to AI Transparency

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bereska-2024-mechanistic-interpretability-ai-safety-review/