Elamur Research⌕
☰
home / papers / 2024 / A Primer on the Inner Workings of Transformer-based Language Models

A Primer on the Inner Workings of Transformer-based Language Models

Paper · 2024 · Javier Ferrando, Gabriele Sarti, Arianna Bisazza et al.

Summary

This survey addresses a gap in interpretability research. Insights into how language models work internally have accumulated quickly but remain scattered. Earlier NLP surveys centred mostly on encoder models such as BERT. Ferrando, Sarti, Bisazza and Costa-jussà (UPC, University of Groningen, FAIR Meta) instead give a compact technical introduction aimed at generative decoder-only (GPT-like) Transformers. They first describe the architecture from the residual stream viewpoint, in which attention and feedforward blocks add their outputs to a shared running representation, and they use one consistent notation throughout. They then sort existing methods into two families. The first, behavior localization, finds which inputs or components drive a prediction. The second, information decoding, works out what learned representations encode and how that content is used. Next comes a catalogue of internal mechanisms found so far, followed by a list of practical tools. The paper contributes no new data or experiments. Its value lies in linking separate lines of research and making the assumptions behind each method explicit, which makes it a useful entry point for anyone analysing modern LMs.

Key points

  • Method taxonomy. The paper organises methods into two groups:
    • Localization (Section 3): input attribution and model component importance.
    • Decoding (Section 4): probing, the linear representation hypothesis with sparse autoencoders (SAEs), and projection into vocabulary space.
  • Known mechanisms. Section 5 reviews findings for attention heads, feedforward blocks, the residual stream and emergent behaviors spanning several components. Section 6 surveys interpretability tooling.
  • Simplifying the math. Attention heads are rewritten as QK (query-key) and OV (output-value) circuits. LayerNorm is approximated as an affine map and folded into the following linear layer. The paper notes that Llama 2 uses RMSNorm instead.
  • FFN interpretation. Following Geva et al. (2021), FFN layers are read as key-value memories. The elementwise nonlinearity creates a privileged basis, which makes individual neurons easier to interpret.
  • SAE training advice (appendix). The appendix covers how to handle "dead" features, through resampling or ghost gradients, and the effect of setting Adam's β1 to 0. Notation tables are also provided (arXiv:2405.00208v3, Oct 2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksInterpretability · 149 works

Cite

Cite as Javier Ferrando, Gabriele Sarti, Arianna Bisazza and Marta R. Costa-jussa. 2024. A Primer on the Inner Workings of Transformer-based Language Models. arXiv:2405.00208.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.00208 · arXiv 2405.00208

@misc{ferrando2024primer,
  title = {{A Primer on the Inner Workings of Transformer-based Language Models}},
  author = {Javier Ferrando, Gabriele Sarti, Arianna Bisazza and Marta R. Costa-jussa},
  year = {2024},
  doi = {10.48550/arxiv.2405.00208},
  eprint = {2405.00208},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.00208}
}

References

436 references, 123 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Abnar and W. Zuidema. Quantifying attention flow in transformers. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, Online, July 2020. Association for Computational Linguistics. do…
  2. R. Achtibat, S. M. V. Hatefi, M. Dreyer, A. Jain, T. Wiegand, S. Lapuschkin, and W. Samek. AttnLRP: Attention-aware layer-wise relevance propagation for transformers, 2024. (p. 7) find
  3. J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 9525–9536, Red Hook, NY, USA, 2018. Curran Associates Inc. (p. 8) find
  4. J. Adebayo, M. Muelly, I. Liccardi, and B. Kim. Debugging tests for model explanations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. (p. 8) find
  5. J. Adebayo, M. Muelly, H. Abelson, and B. Kim. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=xNOVfCCvDpM. (p. 8) ↗
  6. C. Agarwal, S. H. Tanneru, and H. Lakkaraju. Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models. ArXiv, abs/2402.04614, 2024. URL https://api.semanticscholar.org/CorpusID:267523276. (p. 19) — [arXiv:2402.04614](https://arxiv.org/abs/2402.04614) ↗
  7. A. Ahmadian, S. Dash, H. Chen, B. Venkitesh, Z. S. Gou, P. Blunsom, A. Üstün, and S. Hooker. Intriguing properties of quantization at scale. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=IYe8j7Gy8f. (p. 25) ↗
  8. E. Akyurek, T. Bolukbasi, F. Liu, B. Xiong, I. Tenney, J. Andreas, and K. Guu. Towards tracing knowledge in language models back to the training data. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2429–2446, Abu Dhabi, Un… find
  9. E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I. (p. 31) ↗
  10. E. Akyürek, B. Wang, Y. Kim, and J. Andreas. In-context language learning: Architectures and algorithms, 2024. (p. 21) find
  11. G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. Arxiv, 2016. URL https://arxiv.org/abs/1610.01644. (p. 13) ↗
  12. J. Alammar. Ecco: An open source library for the explainability of transformer language models. In H. Ji, J. C. Park, and R. Xia (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing:… find
  13. A. Ali, T. Schnake, O. Eberle, G. Montavon, K.-R. Müller, and L. Wolf. XAI for transformers: Better explanations through conservative propagation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning,… find
  14. A. Ali, I. Zimerman, and L. Wolf. The hidden attention of mamba models, 2024. (p. 29)
  15. K. Amara, R. Sevastjanova, and M. El-Assady. Syntaxshap: Syntax-aware explainability method for text generation. ArXiv, abs/2402.09259, 2024. URL https://api.semanticscholar.org/CorpusID: 267657673. (p. 7) ↗
  16. C. Anil, Y. Wu, A. J. Andreassen, A. Lewkowycz, V. Misra, V. V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. Exploring length generalization in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL h…
  17. Anthropic. Introducing the next generation of claude. 2024. URL https://www.anthropic.com/news/cla ude-3-family. (p. 25) ↗
  18. A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda. Refusal in language models is mediated by a single direction. Arxiv, 2024. URL https://arxiv.org/abs/2406.11717. (p. 15) ↗
  19. A. Arora, D. Jurafsky, and C. Potts. Causalgym: Benchmarking causal interpretability methods on linguistic tasks, 2024. URL https://arxiv.org/abs/2402.12560. (p. 13, 15) ↗
  20. S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483–495, 2018. doi: 10.116 2/tacl_a_00034. URL https://aclanthology.org/Q18-1034. (p. 15) ↗
  21. P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein. A diagnostic study of explainability techniques for text classification. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3256–3274, Online,…
  22. P. Atanasova, O.-M. Camburu, C. Lioma, T. Lukasiewicz, J. G. Simonsen, and I. Augenstein. Faithfulness tests for natural language explanations. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Shor…
  23. G. Attanasio, E. Pastor, C. Di Bonaventura, and D. Nozza. ferret: a framework for benchmarking explainers on transformers. In D. Croce and L. Soldaini (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 256–2… find
  24. M. Avitan, R. Cotterell, Y. Goldberg, and S. Ravfogel. What changed? converting representational interventions to natural language. Arxiv, 2024. URL https://arxiv.org/abs/2402.11355. (p. 32) — [arXiv:2402.11355](https://arxiv.org/abs/2402.11355) ↗
  25. A. Azaria and T. Mitchell. The internal state of an LLM knows when it’s lying. In H. Bouamor, J. Pino, and K. Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967–976, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.fi… — The Internal State of an LLM Knows When It's Lying
  26. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. Arxiv, 2016. URL https://arxiv.org/abs/16 07.06450. (p. 3) ↗
  27. S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46, 07 2015. doi: 10.1371/journal.pone.0130140. URL https://doi.org/10.1371/journal.pone.0130140. (p. 7) — [doi:10.1371/journal.pone.0130140](https://doi.org/10.1371/journal.pone.0130140) ↗
  28. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J.… find
  29. D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In D. Precup and Y. W. Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Ma… find
  30. J. Bastings and K. Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In A. Alishahi, Y. Belinkov, G. Chrupała, D. Hupkes, Y. Pinter, and H. Sajjad (eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting N…
  31. J. Bastings, S. Ebert, P. Zablotskaia, A. Sandholm, and K. Filippova. “will you find these shortcuts?” a protocol for evaluating the faithfulness of input salience methods for text classification. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Metho… find
  32. A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1z-PsR5KX. (p. 23) ↗
  33. D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020. doi: 10.1073/pnas.1907375117. URL https://www.pnas.org/doi/abs/10.1073/pnas.1907375… — [doi:10.1073/pnas.1907375117](https://doi.org/10.1073/pnas.1907375117) ↗
  34. D. Bau, B. C. Wallace, A. Guha, J. Bell, and C. Brodley. National deep inference facility for very large language models (ndif). United States National Science Foundation, 2023. (p. 32) find
  35. Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1): 207–219, March 2022. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7. (p. 14) ↗
  36. Y. Belinkov and J. Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72, 2019. doi: 10.1162/tacl_a_00254. URL https://aclanthology.org/Q19-1004. (p. 14) ↗
  37. Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? In R. Barzilay and M.-Y. Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861–872, Vancouver,…
  38. N. Belrose. Least-squares concept erasure with oracle concept labels. EleutherAI Blog, 2023. URL https://blog.eleuther.ai/oracle-leace/. (p. 14) ↗
  39. N. Belrose. Sparse autoencoders. GitHub repository, 2024. URL https://github.com/EleutherAI/sae. (p. 31) ↗
  40. N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens. Arxiv, 2023a. URL https://arxiv.org/abs/2303.08112. (p. 18, 31) ↗
  41. N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https://openreview.net/forum?id=awIpKpwTwF. (p. 14) ↗
  42. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3(null):1137–1155, mar 2003. ISSN 1532-4435. (p. 2)
  43. L. Bereska and E. Gavves. Mechanistic interpretability for ai safety – a review. ArXiv, 2024. URL https://arxiv.org/abs/2404.14082. (p. 2) ↗
  44. L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". ArXiv, abs/2309.12288, 2023. URL https://api.se manticscholar.org/CorpusID:262083829. (p. 30) ↗
  45. A. Bibal, R. Cardon, D. Alfter, R. Wilkens, X. Wang, T. François, and P. Watrin. Is attention explanation? an introduction to the debate. In S. Muresan, P. Nakov, and A. Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  46. S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal. Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Confer… find
  47. A. Bietti, V. Cabannes, D. Bouchacourt, H. Jegou, and L. Bottou. Birth of a transformer: A memory viewpoint. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 1560–1588. Curran Associates, Inc., 2023. URL… find
  48. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023. (p. 19, 32) ↗
  49. B. Bilodeau, N. Jaques, P. W. Koh, and B. Kim. Impossibility theorems for feature attribution. Proceedings of the National Academy of Sciences, 121(2):e2304406120, 2024. doi: 10.1073/pnas.2304406120. URL https://www.pnas.org/doi/abs/10.1073/pnas.2304406120. (p. 8) — [doi:10.1073/pnas.2304406120](https://doi.org/10.1073/pnas.2304406120) ↗
  50. J. Bloom. Open source sparse autoencoders for all residual stream layers of GPT2 small. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/open-source-sparse-a utoencoders-for-all-residual-stream. (p. 24, 25) ↗
  51. J. Bloom and D. Channin. Saelens. GitHub repository, 2024. URL https://github.com/jbloomAus/SAELe ns. (p. 31) ↗
  52. J. Bloom and J. Lin. Understanding SAE features with the logit lens. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/qykrYY6rXXM7EEs8Q/understanding-sae-features-wit h-the-logit-lens. (p. 25) ↗
  53. T. Bolukbasi, A. Pearce, A. Yuan, A. Coenen, E. Reif, F. Viégas, and M. Wattenberg. An interpretability illusion for bert, 2021. (p. 19) find
  54. Y. Bondarenko, M. Nagel, and T. Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=sbusw6LD41. (p. 22, 25) ↗
  55. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards mon… find
  56. S. Brody, U. Alon, and E. Yahav. On the expressivity role of LayerNorm in transformers’ attention. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 14211–14221, Toronto, Canada, July 2023. Association for Computational Ling… find
  57. D. Brown, N. Vyas, and Y. Bansal. On privileged and convergent bases in neural network representations, 2023. (p. 5) find
  58. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… find
  59. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/pap er_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. (p. 2) ↗
  60. M.-E. Brunet, C. Alkalay-Houlihan, A. Anderson, and R. Zemel. Understanding the origins of bias in word embeddings. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 803–811. P… find
  61. G. Brunner, Y. Liu, D. Pascual, O. Richter, M. Ciaramita, and R. Wattenhofer. On identifiability in transformers. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum ?id=BJg1f6EFDB. (p. 7) ↗
  62. C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs. (p. 14) ↗
  63. B. Bussmann, P. Leask, and N. Nanda. Batchtopk: A simple improvement for topk-saes. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/Nkx6yWZNbAsfvic98/batchtopk-a-simple-i mprovement-for-topk-saes. (p. 17) ↗
  64. N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/distill.00024. URL https://distill.pub/2020/circuits. (p. 11, 26) — [doi:10.23915/distill.00024](https://doi.org/10.23915/distill.00024) ↗
  65. N. Cancedda. Spectral filters, dark signals, and attention sinks, 2024. URL https://arxiv.org/abs/2402.0 9221. (p. 18, 22) ↗
  66. S. Casper, C. Ezell, C. Siegmann, N. Kolt, T. L. Curtis, B. Bucknall, A. A. Haupt, K. Wei, J. Scheurer, M. Hobbhahn, L. Sharkey, S. Krishna, M. von Hagen, S. Alberti, A. Chan, Q. Sun, M. Gerovitch, D. Bau, M. Tegmark, D. Krueger, and D. Hadfield-Menell. Black-box access is insufficient for rigorous… — [arXiv:2401.14446](https://arxiv.org/abs/2401.14446) find
  67. S. CH-Wang, B. V. Durme, J. Eisner, and C. Kedzie. Do androids know they’re only dreaming of electric sheep?, 2023. URL https://arxiv.org/abs/2312.17249v1. (p. 14, 27) ↗
  68. L. Chan, A. Garriga-Alonso, N. Goldwosky-Dill, R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/cau sal-s… ↗
  69. H. Chefer, S. Gur, and L. Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, June 2021. (p. 7) find
  70. A. Chen, R. Shwartz-Ziv, K. Cho, M. L. Leavitt, and N. Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=MO5PiKHELW. (p. 20) ↗
  71. C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024b. URL — INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection https://openreview.net/forum?id=Zj12nzlQbz. (p. 27) ↗
  72. H. Chen, C. Vondrick, and C. Mao. Selfie: Self-interpretation of large language model embeddings, 2024c. URL https://arxiv.org/abs/2403.10949. (p. 18, 32) — [arXiv:2403.10949](https://arxiv.org/abs/2403.10949) ↗
  73. S. Chen, M. Xiong, J. Liu, Z. Wu, T. Xiao, S. Gao, and J. He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation, 2024d. (p. 27) find
  74. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… — PaLM: Scaling Language Modeling with Pathways
  75. A. G. Chowdhury, M. M. Islam, V. Kumar, F. H. Shezan, V. Kumar, V. Jain, and A. Chadha. Breaking down the defenses: A comparative survey of attacks on large language models, 2024. (p. 19) find
  76. Y.-S. Chuang, Y. Xie, H. Luo, Y. Kim, J. R. Glass, and P. He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Th6NyL07na. (p. 29) ↗
  77. B. Chughtai, A. Cooney, and N. Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms, 2024. URL https://www.arxiv.org/abs/2402.07321. (p. 21, 29) — [arXiv:2402.07321](https://arxiv.org/abs/2402.07321) ↗
  78. K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does BERT look at? an analysis of BERT’s attention. In T. Linzen, G. Chrupała, Y. Belinkov, and D. Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286, Florence, Ita… find
  79. T. Conerly, A. Templeton, T. Bricken, J. Marcus, and T. Henighan. Circuits updates - april 2024. update on how we train saes. find
  80. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/april-update/index.html. (p. 57) ↗
  81. A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 16318–163… find
  82. A. Cooney. CircuitVis, December 2022. URL https://github.com/alan-cooney/CircuitsVis. (p. 31) ↗
  83. A. Cooney. Sparse autoencoder. GitHub repository, 2023. URL https://github.com/ai-safety-foundat ion/sparse_autoencoder. (p. 31) ↗
  84. G. M. Correia, V. Niculae, and A. F. T. Martins. Adaptively sparse transformers. In K. Inui, J. Jiang, V. Ng, and X. Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC…
  85. M. Costa-jussà, E. Smith, C. Ropers, D. Licht, J. Maillard, J. Ferrando, and C. Escolano. Toxicity in multilingual machine translation at scale. In H. Bouamor, J. Pino, and K. Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9570–9586, Singapore, December 2023.… find
  86. I. Covert, S. Lundberg, and S.-I. Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1–90, 2021. URL http://jmlr.org/papers/v22/20-1 316.html. (p. 7) ↗
  87. J. Crabbé and M. van der Schaar. Evaluating the robustness of interpretability methods through explanation invariance and equivariance. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=5UwnKSgY6u. (p. 8) ↗
  88. R. Csordás, S. van Steenkiste, and J. Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD. (p. 11) ↗
  89. H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. Arxiv, 2023. URL https://arxiv.org/abs/2309.08600. (p. 16, 25) ↗
  90. D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493–8502, Dublin, Ireland,… find
  91. D. Dale, E. Voita, L. Barrault, and M. R. Costa-jussà. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Associatio…
  92. D. Dale, E. Voita, J. Lam, P. Hansanti, C. Ropers, E. Kalbassi, C. Gao, L. Barrault, and M. Costa-jussà. HalOmi: A manually annotated benchmark for multilingual hallucination and omission detection in machine translation. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference… find
  93. F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, A. Bau, and J. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intellige… — [doi:10.1609/aaai.v33i01.33016309](https://doi.org/10.1609/aaai.v33i01.33016309) find
  94. P. A. Daniel Johnson. Penzai. GitHub repository, 2024. URL https://github.com/google-deepmind/pe nzai. (p. 31) ↗
  95. J. Dao, Y.-T. Lau, C. Rager, and J. Janiak. An adversarial example for direct logit attribution: Memory management in gelu-4l, 2023. (p. 25) find
  96. G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16124–16170, Toronto, Canada, July 2023. Asso…
  97. T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO 3LLiJ1. (p. 25) ↗
  98. Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 933–941. JMLR.org, 2017. (p. 16)
  99. N. De Cao, M. S. Schlichtkrull, W. Aziz, and I. Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),… find
  100. N. De Cao, W. Aziz, and I. Titov. Editing factual knowledge in language models. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491–6506, Online and Punta Cana, Dominican Republic, November 202… find
  101. N. De Cao, L. Schmid, D. Hupkes, and I. Titov. Sparse interventions in language models with differentiable masking. In J. Bastings, Y. Belinkov, Y. Elazar, D. Hupkes, N. Saphra, and S. Wiegreffe (eds.), Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for N… find
  102. B. Deiseroth, M. Deb, S. Weinbach, M. Brack, P. Schramowski, and K. Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems… find
  103. M. Denil, A. Demiraj, and N. de Freitas. Extraction of salient sentences from labelled documents. Arxiv, 2015. URL https://arxiv.org/abs/1412.6815. (p. 7) — [arXiv:1412.6815](https://arxiv.org/abs/1412.6815) ↗
  104. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30318–30332. Curran Associates, In… find
  105. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Huma…
  106. J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4…
  107. K. Dhamdhere, M. Sundararajan, and Q. Yan. How important is a neuron. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SylKoo0cKm. (p. 12) ↗
  108. S. Dhanorkar, C. T. Wolf, K. Qian, A. Xu, L. Popa, and Y. Li. Who needs to know what, when?: Broadening the explainable ai (xai) design space by looking at explanations across the ai lifecycle. In Proceedings of the 2021 ACM Designing Interactive Systems Conference, DIS ’21, pp. 1591–1602, New York,… — [doi:10.1145/3461778.3462131](https://doi.org/10.1145/3461778.3462131) find
  109. A. Y. Din, T. Karidi, L. Choshen, and M. Geva. Jump to conclusions: Short-cutting transformers with linear transformations. Arxiv, 2023. URL https://arxiv.org/abs/2303.09435. (p. 18) ↗
  110. S. Ding and P. Koehn. Evaluating saliency methods for neural language models. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association f… find
  111. F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning, 2017. URL https://arxiv.org/abs/1702.08608. (p. 32) ↗
  112. N. Durrani, F. Dalvi, and H. Sajjad. Discovering salient neurons in deep nlp models. Journal of Machine Learning Research, 24(362):1–40, 2023. URL http://jmlr.org/papers/v24/23-0074.html. (p. 23) ↗
  113. Y. Elazar, S. Ravfogel, A. Jacovi, and Y. Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 03 2021. ISSN 2307-387X. doi: 10.1162/tacl_a_00359. URL https://doi.org/10.1162/tacl_a_00359. (p. 14, 15… ↗
  114. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematic… find
  115. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. Garcon. Tran… find
  116. N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E.… find
  117. N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022b. URL https://transformer-circuits.pub/2022… ↗
  118. N. Elhage, R. Lasenby, and C. Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/privileged-basis/index.html. (p. 25) ↗
  119. J. Enguehard. Sequential integrated gradients: a simple but effective method for explaining language models. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 7555–7565, Toronto, Canada, July 2023. Association for Computatio… find
  120. N. B. Erichson, Z. Yao, and M. W. Mahoney. Jumprelu: A retrofit defense strategy for adversarial attacks. ArXiv, 2019. URL https://arxiv.org/abs/1904.03750. (p. 17) — [arXiv:1904.03750](https://arxiv.org/abs/1904.03750) ↗
  121. K. Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In K. Inui, J. Jiang, V. Ng, and X. Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… find
  122. K. Ethayarajh and D. Jurafsky. Attention flows are shapley value explanations. In C. Zong, F. Xia, W. Li, and R. Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  123. P. Fantozzi and M. Naldi. The explainability of transformers: Current status and directions. Computers, 13(4), 2024. ISSN 2073-431X. doi: 10.3390/computers13040092. URL https://www.mdpi.com/2073-431X/13/4/92. (p. 6) — [doi:10.3390/computers13040092](https://doi.org/10.3390/computers13040092) ↗
  124. N. Feldhus, L. Hennig, M. D. Nasert, C. Ebert, R. Schwarzenberg, and S. Möller. Saliency map verbalization: Comparing feature importance representations from model-free and instruction-based methods. In B. Dalvi Mishra, G. Durrett, P. Jansen, D. Neves Ribeiro, and J. Wei (eds.), Proceedings of the 1… find
  125. J. Ferrando and M. R. Costa-jussà. Attention weights in transformer NMT fail aligning words between sequences but largely explain model predictions. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 434–443, Punta… find
  126. J. Ferrando and M. R. Costa-jussà. On the similarity of circuits across languages: a case study on the subject-verb agreement task. arXiv, 2024. URL https://arxiv.org/abs/2410.06496. (p. 27) ↗
  127. J. Ferrando and E. Voita. Information flow routes: Automatically interpreting language models at scale. Arxiv, 2024. URL https://arxiv.org/abs/2403.00824. (p. 4, 12, 20, 21, 22) ↗
  128. J. Ferrando, G. I. Gállego, B. Alastruey, C. Escolano, and M. R. Costa-jussà. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods i… find
  129. J. Ferrando, G. I. Gállego, and M. R. Costa-jussà. Measuring the mixing of contextual information in the transformer. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8698–8714, Abu Dhabi, United Arab Emirat… find
  130. J. Ferrando, G. I. Gállego, I. Tsiamas, and M. R. Costa-jussà. Explaining how transformers use context to build predictions. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5486–… find
  131. C. Fierro and A. Søgaard. Factual consistency of multilingual pretrained language models. In S. Muresan, P. Nakov, and A. Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3046–3052, Dublin, Ireland, May 2022. Association for Computational Linguistics. do… find
  132. J. Fiotto-Kaufman. nnsight: The package for interpreting and manipulating the internals of deep learned models. , 2024. URL https://github.com/JadenFiotto-Kaufman/nnsight. (p. 30) ↗
  133. M. Fomicheva, S. Sun, L. Yankovskaya, F. Blain, F. Guzmán, M. Fishel, N. Aletras, V. Chaudhary, and L. Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555, 2020. doi: 10.1162/tacl_a_00330. URL https://aclan…
  134. D. Friedman, A. Wettig, and D. Chen. Learning transformer programs. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023. URL https://openreview.net/forum?id=Pe9WxkN8Ff. (p. 31) ↗
  135. L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.04093v1. (p. 17) ↗
  136. A. Geiger, K. Richardson, and C. Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In A. Alishahi, Y. Belinkov, G. Chrupała, D. Hupkes, Y. Pinter, and H. Sajjad (eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpret…
  137. A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 9574–9586. Curran Associates, Inc., 2021. URL https://proceedings.neurips… ↗
  138. A. Geiger, Z. Wu, H. Lu, J. Rozner, E. Kreiss, T. Icard, N. D. Goodman, and C. Potts. Inducing causal structure for interpretable neural networks, 2022. URL https://arxiv.org/abs/2112.00826. (p. 12, 30) ↗
  139. A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation, 2023a. URL https://arxiv.org/abs/2301.04709. (p. 12, 30) ↗
  140. A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations, 2023b. URL https://arxiv.org/abs/2303.02536. (p. 11, 13) ↗
  141. G. Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari,… find
  142. M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484–5495, Online and Punta Cana, Dominican… find
  143. M. Geva, A. Caciularu, G. Dar, P. Roit, S. Sadde, M. Shlain, B. Tamir, and Y. Goldberg. LM-debugger: An interactive tool for inspection and intervention in transformer-based language models. In W. Che and E. Shutova (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language… find
  144. M. Geva, A. Caciularu, K. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30–45, Abu D…
  145. M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in autoregressive language models. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216–12235, Singapore, December… find
  146. A. Ghandeharioun, A. Caciularu, A. Pearce, L. Dixon, and M. Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. Arxiv, 2024. URL https://arxiv.org/abs/2401 .06102v2. (p. 18) ↗
  147. N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora. Localizing model behavior with path patching. Arxiv, 2023. URL https://arxiv.org/abs/2304.05969. (p. 12) ↗
  148. R. Gould, E. Ong, G. Ogden, and A. Conmy. Successor heads: Recurring, interpretable attention heads in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openre view.net/forum?id=kvcbV8KQsi. (p. 21) find
  149. D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, V. Pya… find
  150. R. B. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukovsiut.e, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. Bowman. Studying large language model generalization with influence functions. ArXiv, abs/2308.03296, 2023. URL…
  151. A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. (p. 28) find
  152. J.-C. Gu, H. Xu, J.-Y. Ma, P. Lu, Z.-H. Ling, K. wei Chang, and N. Peng. Model editing can hurt general abilities of large language models. ArXiv, abs/2401.04700, 2024. URL https://api.semanticscholar.org/CorpusID:266899568. (p. 30) — [arXiv:2401.04700](https://arxiv.org/abs/2401.04700) ↗
  153. C. Guerner, A. Svete, T. Liu, A. Warstadt, and R. Cotterell. A geometric notion of causal probing, 2023. (p. 11) find
  154. N. M. Guerreiro, P. Colombo, P. Piantanida, and A. Martins. Optimal transport for unsupervised hallucination detection in neural machine translation. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1… find
  155. N. M. Guerreiro, E. Voita, and A. Martins. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In A. Vlachos and I. Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, p…
  156. A. Gupta, G. Boleda, M. Baroni, and S. Padó. Distributional vectors encode referential attributes. In L. Màrquez, C. Callison-Burch, and J. Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 12–21, Lisbon, Portugal, September 2015. Association for… find
  157. A. Gupta, A. Rao, and G. K. Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. ArXiv, abs/2401.07453, 2024a. URL https://api.semanticscholar.org/CorpusID:266999650. (p. 30) ↗
  158. A. Gupta, D. Sajnani, and G. Anumanchipalli. A unified framework for model editing, 2024b. (p. 30) find
  159. W. Gurnee. Sae reconstruction errors are (empirically) pathological. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/rZPiuFxESMxCDHe4B/sae-reconstruction-errors-are-empir ically-pathological. (p. 16) ↗
  160. W. Gurnee and M. Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jE8xbmvFin. (p. 23) ↗
  161. W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=JYs1R9IMJr. (p. 15, 23, 24) ↗
  162. W. Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, and D. Bertsimas. Universal neurons in gpt2 language models, 2024. (p. 22, 23, 24, 25) find
  163. K. Guu, A. Webson, E. Pavlick, L. Dixon, I. Tenney, and T. Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. Arxiv, 2023. URL https://arxiv.org/abs/23 03.08114. (p. 8) ↗
  164. Z. Hammoudeh and D. Lowd. Training data influence analysis and estimation: A survey, 2022. (p. 8) find
  165. X. Han, B. C. Wallace, and Y. Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5553–5563, On…
  166. Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. (p. 31)
  167. M. Hanna, O. Liu, and A. Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 76033–76060…
  168. M. Hanna, S. Pezzelle, and Y. Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms, 2024. URL https://arxiv.org/abs/2403.17806. (p. 12, 27) ↗
  169. P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processin… find
  170. A. Haviv, I. Cohen, J. Gidron, R. Schuster, Y. Goldberg, and M. Geva. Understanding transformer memorization recall through idioms. In A. Vlachos and I. Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 248–264, Dubrov…
  171. Z. He, X. Ge, Q. Tang, T. Sun, Q. Cheng, and X. Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. ArXiv, abs/2402.12201, 2024. URL https://api.semanticscholar.org/CorpusID:267751496. (p. 16) — [arXiv:2402.12201](https://arxiv.org/abs/2402.12201) ↗
  172. S. Heimersheim and J. Janiak. A circuit for python docstrings in a 4-layer attention-only transformer. AI Alignment Forum, 2023. URL https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/a-cir cuit-for-python-docstrings-in-a-4-layer-attention-only. (p. 10, 11, 22, 26) ↗
  173. S. Heimersheim and N. Nanda. How to use and interpret activation patching. Arxiv, 2024. URL https://arxiv.org/abs/2404.15255. (p. 10) — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) ↗
  174. S. Heimersheim and A. Turner. Residual stream norms grow exponentially over the forward pass. AI Alignment Forum, 2023. URL https://www.alignmentforum.org/posts/8mizBCm3dyc432nK8/residual-strea m-norms-grow-exponentially-over-the-forward. (p. 24) ↗
  175. R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. Arxiv, 2023. URL https://arxiv.org/abs/2310.15916. (p. 25, 26) ↗
  176. E. Hernandez, A. S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y. Belinkov, and D. Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=w7LU2s14kE. (p. 29) ↗
  177. J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In K. Inui, J. Jiang, V. Ng, and X. Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… find
  178. J. Hewitt and C. D. Manning. A structural probe for finding syntax in word representations. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long…
  179. J. Hewitt, J. Thickstun, C. Manning, and P. Liang. Backpack language models. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9103–9125, Toronto, Canada, July 2023. Association fo… find
  180. A. Himmi, G. Staerman, M. Picot, P. Colombo, and N. M. Guerreiro. Enhanced hallucination detection in neural machine translation through simple detector aggregation, 2024. (p. 28) find
  181. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre. An empirical analysis o… find
  182. A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 70…
  183. B. Hoover, H. Strobelt, and S. Gehrmann. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In A. Celikyilmaz and T.-H. Wen (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187–196, Onlin… find
  184. P. M. Htut, J. Phang, S. Bordia, and S. R. Bowman. Do attention heads in bert track syntactic dependencies? Arxiv, 2019. URL https://arxiv.org/abs/1911.12246. (p. 20) ↗
  185. J. Y.-C. Hu, P.-H. Chang, R. Luo, H.-Y. Chen, W. Li, W.-P. Wang, and H. Liu. Outlier-efficient hopfield layers for large transformer-based models. In Forty-first International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2404.03828. (p. 25) — [arXiv:2404.03828](https://arxiv.org/abs/2404.03828) ↗
  186. J. Huang, A. Geiger, K. D’Oosterlinck, Z. Wu, and C. Potts. Rigorously assessing natural language explanations of neurons. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,…
  187. J. Huang, Z. Wu, C. Potts, M. Geva, and A. Geiger. Ravel: Evaluating interpretability methods on disentangling language model representations, 2024a. URL https://arxiv.org/abs/2402.17700. (p. 13, 15) ↗
  188. Y. Huang, S. Hu, X. Han, Z. Liu, and M. Sun. Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition, 2024b. (p. 27) find
  189. N. Hudson, J. G. Pauloski, M. Baughman, A. Kamatar, M. Sakarvadia, L. Ward, R. Chard, A. Bauer, M. Levental, W. Wang, W. Engler, O. P. Skelly, B. Blaiszik, R. Stevens, K. Chard, and I. Foster. Trillion parameter ai serving infrastructure for scientific discovery: A survey and vision. Arxiv, 2024. UR… — [arXiv:2402.03480](https://arxiv.org/abs/2402.03480) find
  190. D. Hupkes, S. Veldhoen, and W. Zuidema. Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61(1): 907–926, 2018. ISSN 1076-9757. (p. 14) find
  191. S. Jain, R. Kirk, E. S. Lubana, R. P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, and D. Krueger. What happens when you fine-tuning your model? mechanistic analysis of procedurally generated tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevie… find
  192. S. Jain and B. C. Wallace. Attention is not Explanation. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543–3556, M…
  193. S. Jastrz˛ebski, D. Arpit, N. Ballas, V. Verma, T. Che, and Y. Bengio. Residual connections encourage iterative inference, 2018. (p. 17) find
  194. T. D. L. T. Jeffrey Wu, Leo Gao. Sparse autoencoders. GitHub repository, 2024. URL https://github.com/openai/sparse_autoencoder. (p. 31) ↗
  195. A. Jermyn and A. Templeton. Circuits updates - jnauary 2024. ghost grads: An improvement on resampling. find
  196. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/jan-update/in dex.html. (p. 57) ↗
  197. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), mar 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https://doi.org/10.1145/3571730. (p. 27) ↗
  198. Y. Jiang, G. Rajendran, P. Ravikumar, B. Aragam, and V. Veitch. On the origins of linear representations in large language models, 2024. (p. 14) find
  199. S. Joseph. Vit prisma: A mechanistic interpretability library for vision transformers. GitHub repository, 2023. URL https://github.com/soniajoseph/vit-prisma. (p. 31) ↗
  200. G. Kamradt. Needle in a haystack - pressure testing llms. Github Repository, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack. (p. 29) ↗
  201. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361. (p. 2) ↗
  202. A. Karvonen, B. Wright, C. Rager, R. Angell, J. Brinkmann, L. Smith, C. M. Verdun, D. Bau, and S. Marks. Measuring progress in dictionary learning for language model interpretability with board game models, 2024. URL https://arxiv.org/abs/2408.00113. (p. 16) — [arXiv:2408.00113](https://arxiv.org/abs/2408.00113) ↗
  203. S. Katz and Y. Belinkov. VISIT: Visualizing and interpreting the semantic information flow of transformers. In H. Bouamor, J. Pino, and K. Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14094–14113, Singapore, December 2023. Association for Computational Ling… find
  204. S. Katz, Y. Belinkov, M. Geva, and L. Wolf. Backward lens: Projecting language model gradients into the vocabulary space, 2024. (p. 18) find
  205. B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of P… find
  206. C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Sparse autoencoders work on attention layer outputs. AI Alignment Forum, 2024a. URL https://www.alignmentforum.org/posts/DtdzGwFh9dCfsekZZ. (p. 22) ↗
  207. C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Attention saes scale to gpt-2 small. AI Alignment Forum, 2024b. URL https://www.alignmentforum.org/posts/FSTRedtjuHa4Gfdbr/attention-sae s-scale-to-gpt-2-small. (p. 17, 22) ↗
  208. G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Attention is not only a weight: Analyzing transformers with vector norms. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057–7075, Online, Novem…
  209. G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Incorporating Residual and Normalization Layers into Analysis of Masked Language Models. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4547… find
  210. G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Transformer language models handle word frequency in prediction head. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 4523–4535, Toronto, Canada, July 2023. Association… find
  211. G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Analyzing feed-forward blocks in transformers through the lens of attention map. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mYWsyTuiRp. (p. 8) ↗
  212. P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In D. Precup and Y. W. Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885–1894. PMLR, 06–11 Aug 2017. URL https://pr oce…
  213. A. Köhn. What’s in an embedding? analyzing word embeddings through multilingual evaluation. In L. Màrquez, C. Callison-Burch, and J. Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2067–2073, Lisbon, Portugal, September 2015. Association for Com… find
  214. N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz-Richardson. Captum: A unified and generic model interpretability library for pytorch. Arxiv, 2020. URL https://arxiv.org/abs/2009.07896. (p. 30) — [arXiv:2009.07896](https://arxiv.org/abs/2009.07896) ↗
  215. O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the dark secrets of BERT. In K. Inui, J. Jiang, V. Ng, and X. Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
  216. O. Kovaleva, S. Kulshreshtha, A. Rogers, and A. Rumshisky. BERT busters: Outlier dimensions that disrupt transformers. In C. Zong, F. Xia, W. Li, and R. Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3392–3405, Online, August 2021. Association for Com…
  217. J. Kramár, T. Lieberum, R. Shah, and N. Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URL https://arxiv.org/abs/2403.00745. (p. 10, 12) ↗
  218. S. Krishna, T. Han, A. Gu, S. Wu, S. Jabbari, and H. Lakkaraju. The disagreement problem in explainable machine learning: A practitioner’s perspective. ArXiv, 2024. URL https://arxiv.org/abs/2202.01602. (p. 8) — [arXiv:2202.01602](https://arxiv.org/abs/2202.01602) ↗
  219. R. Krzyzanowski, C. Kissane, A. Conmy, and N. Nanda. We inspected every head in GPT-2 small using saes so you don’t have to. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/xme geW5mqiBsvoaim/we-inspected-every-head-in-gpt-2-small-using-saes-so-you-don. (p. 22) ↗
  220. Y. Kwon, E. Wu, K. Wu, and J. Zou. Datainf: Efficiently estimating data influence in loRA-tuned LLMs and diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=9m02ib92Wz. (p. 8) ↗
  221. V. Lal, A. Ma, E. Aflalo, P. Howard, A. Simoes, D. Korat, O. Pereg, G. Singer, and M. Wasserblat. InterpreT: An interactive visualization tool for interpreting transformers. In D. Gkatzia and D. Seddah (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computat… find
  222. A. Langedijk, H. Mohebbi, G. Sarti, W. Zuidema, and J. Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. ArXiv, abs/2310.03686, 2023. URL https://api.semanticscholar.org/CorpusID:263671583. (p. 17) ↗
  223. T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. E. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukovsiut.e, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. D. Maxwell, T. Telleen-Lawton, T… — Measuring Faithfulness in Chain-of-Thought Reasoning
  224. K. Leino, S. Sen, A. Datta, M. Fredrikson, and L. Li. Influence-directed explanations for deep convolutional networks. In 2018 IEEE International Test Conference (ITC), pp. 1–8, 2018. doi: 10.1109/TEST.2018.86 24792. (p. 12) — [doi:10.1109/test.2018.86](https://doi.org/10.1109/test.2018.86) find
  225. M. A. Lepori, T. Serre, and E. Pavlick. Uncovering intermediate variables in transformers using circuit probing, 2023. (p. 13) find
  226. J. Li, X. Chen, E. Hovy, and D. Jurafsky. Visualizing and understanding neural models in NLP. In K. Knight, A. Nenkova, and O. Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681–691, S…
  227. J. Li, W. Monroe, and D. Jurafsky. Understanding neural networks through representation erasure, 2017. (p. 7) find
  228. K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openreview.net/forum?id=aLLuYpn83y. (p. 27) ↗
  229. X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis… find
  230. Z. Li, N. Zhang, Y. Yao, M. Wang, X. Chen, and H. Chen. Unveiling the pitfalls of knowledge editing for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=fNktD3ib16. (p. 30) ↗
  231. Q. V. Liao, D. Gruen, and S. Miller. Questioning the ai: Informing design practices for explainable ai user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pp. 1–15, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145036708… — [doi:10.1145/3313831.3376590](https://doi.org/10.1145/3313831.3376590) find
  232. T. Lieberum, M. Rahtz, J. Kramár, N. Nanda, G. Irving, R. Shah, and V. Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. Arxiv, 2023. URL https://arxiv.org/abs/2307.09458. (p. 10, 12) ↗
  233. T. Lieberum, S. Rajamanoharan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramár, A. Dragan, R. Shah, and N. Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. ArXiv, 2024. URL https://arxiv.org/abs/2408.05147. (p. 17) ↗
  234. J. Lin and J. Bloom. Announcing neuronpedia: Platform for accelerating research into sparse autoencoders. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/BaEQoxHhWPrkinmxd/annou ncing-neuronpedia-platform-for-accelerating-research. (p. 17, 31) ↗
  235. Y. Lin, Y. C. Tan, and R. Frank. Open sesame: Getting inside BERT’s linguistic knowledge. In T. Linzen, G. Chrupała, Y. Belinkov, and D. Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 241–253, Florence, Italy, August 2019. Ass…
  236. D. Lindner, J. Kramar, S. Farquhar, M. Rahtz, T. McGrath, and V. Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 37876–37899. C… find
  237. Z. C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57, jun 2018. ISSN 1542-7730. doi: 10.1145/3236386.3241340. URL https://doi.org/10.1145/3236386.3241340. (p. 6) — [doi:10.1145/3236386.3241340](https://doi.org/10.1145/3236386.3241340) ↗
  238. N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith. Linguistic knowledge and transferability of contextual representations. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic…
  239. Q. Liu, Y. Chai, S. Wang, Y. Sun, K. Wang, and H. Wu. On training data influence of gpt models. Arxiv, 2024. URL https://arxiv.org/abs/2404.07840. (p. 8) ↗
  240. L. Longo, M. Brcic, F. Cabitza, J. Choi, R. Confalonieri, J. D. Ser, R. Guidotti, Y. Hayashi, F. Herrera, A. Holzinger, R. Jiang, H. Khosravi, F. Lecue, G. Malgieri, A. Páez, W. Samek, J. Schneider, T. Speith, and S. Stumpf. Explainable artificial intelligence (xai) 2.0: A manifesto of open challeng… — [doi:10.1016/j.inffus.2024.102301](https://doi.org/10.1016/j.inffus.2024.102301) find
  241. M. Loog, T. Viering, A. Mey, J. H. Krijthe, and D. M. J. Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117(20):10625–10626, 2020. doi: 10.1073/pnas.2001875117. URL https://www.pnas.org/doi/abs/10.1073/pnas.2001875117. (p. 27) — [doi:10.1073/pnas.2001875117](https://doi.org/10.1073/pnas.2001875117) ↗
  242. G. Lopardo, F. Precioso, and D. Garreau. Attention meets post-hoc interpretability: A mathematical perspective. ArXiv, 2024. URL https://arxiv.org/abs/2402.03485. (p. 7) — [arXiv:2402.03485](https://arxiv.org/abs/2402.03485) ↗
  243. S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedin…
  244. Z. Luo, A. Kulmizev, and X. Mao. Positional artefacts propagate through masked language model embeddings. In C. Zong, F. Xia, W. Li, and R. Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural L… find
  245. A. Lv, K. Zhang, Y. Chen, Y. Wang, L. Liu, J.-R. Wen, J. Xie, and R. Yan. Interpreting key mechanisms of factual recall in transformer-based language models. Computing Research Repository, arXiv:2403.19521, 2024. URL https://arxiv.org/abs/2403.19521. (p. 21, 29) — [arXiv:2403.19521](https://arxiv.org/abs/2403.19521) ↗
  246. Q. Lyu, M. Apidianaki, and C. Callison-Burch. Towards Faithful Model Explanation in NLP: A Survey. Computational Linguistics, pp. 1–70, 01 2024. ISSN 0891-2017. doi: 10.1162/coli_a_00511. URL https://doi.org/10.1162/coli_a_00511. (p. 2) ↗
  247. M. MacDiarmid, T. Maxwell, N. Schiefer, J. Mu, J. Kaplan, D. Duvenaud, S. Bowman, A. Tamkin, E. Perez, M. Sharma, C. Denison, and E. Hubinger. Simple probes can catch sleeper agents. Anthropic, 2024. URL https://www.anthropic.com/news/probes-catch-sleeper-agents. (p. 14) ↗
  248. A. Madsen, S. Reddy, and S. Chandar. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55(8), 2022. ISSN 0360-0300. doi: 10.1145/3546577. URL https://doi.org/10.1145/354657 7. (p. 2, 6) ↗
  249. A. Madsen, S. Chandar, and S. Reddy. Are self-explanations from large language models faithful? ArXiv, abs/2401.07927, 2024. URL https://api.semanticscholar.org/CorpusID:266999774. (p. 19) ↗
  250. A. Makelov, G. Lange, A. Geiger, and N. Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=Ebt7JgMHv1. (p. 12) ↗
  251. A. Makelov, G. Lange, and N. Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control, 2024b. URL https://arxiv.org/abs/2405.08366. (p. 16) — [arXiv:2405.08366](https://arxiv.org/abs/2405.08366) ↗
  252. A. Makhzani and B. Frey. k-sparse autoencoders, 2014. URL https://arxiv.org/abs/1312.5663. (p. 17) — [arXiv:1312.5663](https://arxiv.org/abs/1312.5663) ↗
  253. S. Marks and A. Mueller. Dictionary learning. GitHub repository, 2023. URL https://github.com/saprm arks/dictionary_learning. (p. 31) ↗
  254. S. Marks and M. Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2023. URL https://arxiv.org/abs/2310.06824. (p. 15) ↗
  255. S. Marks, C. Rager, E. J. Michaud, Y. Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Computing Research Repository, arXiv:2403.19647, 2024. URL https://arxiv.org/abs/2403.19647. (p. 12, 16, 32) ↗
  256. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In A. Korhonen, D. Traum, and L. Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428–3448, Florence, Italy, J…
  257. C. McDougall. Six (and a half) intuitions for SVD. Callum McDougall Blog, 2023. URL https://www.perf ectlynormal.co.uk/blog-svd. (p. 18) ↗
  258. C. McDougall and J. Bloom. Sae-vis: Announcement post. LessWrong, 2024. URL https://www.lesswron g.com/posts/nAhy6ZquNY7AD3RkD/sae-vis-announcement-post-1. (p. 31) ↗
  259. C. McDougall, A. Conmy, C. Rushing, T. McGrath, and N. Nanda. Copy suppression: Comprehensively understanding an attention head. Arxiv, 2023. URL https://arxiv.org/abs/2310.04625. (p. 21) ↗
  260. T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, M. Wattenberg, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022. doi: 10.1073/pnas.2206625119. URL https://www.pnas.org/doi/abs/1… — [doi:10.1073/pnas.2206625119](https://doi.org/10.1073/pnas.2206625119) ↗
  261. T. McGrath, M. Rahtz, J. Kramar, V. Mikulik, and S. Legg. The hydra effect: Emergent self-repair in language model computations. Arxiv, 2023. URL https://arxiv.org/abs/2307.15771. (p. 9, 12) ↗
  262. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17359–17372. Curran Associates, Inc., 2022. URL https://proc… find
  263. K. Meng, A. S. Sharma, A. J. Andonian, Y. Belinkov, and D. Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MkbcAHIYgyS. (p. 30) ↗
  264. W. Merrill, V. Ramanujan, Y. Goldberg, R. Schwartz, and N. A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural La… find
  265. W. Merrill, N. Tsilivis, and A. Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. ArXiv, abs/2303.11873, 2023. URL https://api.semanticscholar.org/CorpusID: 257636667. (p. 27) — [arXiv:2303.11873](https://arxiv.org/abs/2303.11873) ↗
  266. J. Merullo, C. Eickhoff, and E. Pavlick. A mechanism for solving relational tasks in transformer language models, 2023. URL https://arxiv.org/abs/2305.16130. (p. 29) — [arXiv:2305.16130](https://arxiv.org/abs/2305.16130) ↗
  267. J. Merullo, C. Eickhoff, and E. Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=fpoAYV6Wsk. (p. 27) ↗
  268. P. Michel, O. Levy, and G. Neubig. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/pa… ↗
  269. T. Mickus, D. Paperno, and M. Constant. How to dissect a Muppet: The structure of transformer embedding spaces. Transactions of the Association for Computational Linguistics, 10:981–996, 2022. doi: 10.1162/ta cl_a_00501. URL https://aclanthology.org/2022.tacl-1.57. (p. 5) ↗
  270. V. Miglani, A. Yang, A. Markosyan, D. Garcia-Olano, and N. Kokhlikyan. Using captum to explain generative language models. In L. Tan, D. Milajevs, G. Chauhan, J. Gwinnup, and E. Rippeth (eds.), Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp. 1… find
  271. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, I…
  272. B. Millidge and S. Black. The singular value decompositions of transformer weight matrices are highly interpretable. AI Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/mkbGjzxD8d8Xq KHzA/the-singular-value-decompositions-of-transformer-weight. (p. 18) ↗
  273. B. Millidge and E. Winsor. Basic facts about language model internals. AI Alignment Forum, 2023. URL https://www.alignmentforum.org/posts/PDLfpRwSynu73mxGw/basic-facts-about-language-mod el-internals-1. (p. 24) ↗
  274. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large language models: A survey, 2024. URL https://arxiv.org/abs/2402.06196. (p. 27) ↗
  275. E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id=0DcZxeWf OPt. (p. 30) ↗
  276. E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn. Memory-based model editing at scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Re… find
  277. A. Modarressi, M. Fayyaz, Y. Yaghoobzadeh, and M. T. Pilehvar. GlobEnc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the… find
  278. A. Modarressi, M. Fayyaz, E. Aghazadeh, Y. Yaghoobzadeh, and M. T. Pilehvar. DecompX: Explaining transformers decisions by propagating token decomposition. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo… find
  279. H. Mohebbi, W. Zuidema, G. Chrupała, and A. Alishahi. Quantifying context mixing in transformers. In A. Vlachos and I. Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3378–3400, Dubrovnik, Croatia, May 2023. Associat… find
  280. R. Molina. Traveling words: A geometric interpretation of transformers. Arxiv, 2023. URL https://arxiv.org/abs/2309.07315. (p. 18) — [arXiv:2309.07315](https://arxiv.org/abs/2309.07315) ↗
  281. G. Monea, M. Peyrard, M. Josifoski, V. Chaudhary, J. Eisner, E. Kıcıman, H. Palangi, B. Patra, and R. West. A glitch in the matrix? locating and detecting language model grounding with fakepedia, 2024. URL https://arxiv.org/abs/2312.02073. (p. 29) ↗
  282. D. Mossing, S. Bills, H. Tillman, T. Dupré la Tour, N. Cammarata, L. Gao, J. Achiam, C. Yeh, J. Leike, J. Wu, and W. Saunders. Transformer debugger. https://github.com/openai/transformer-debugger, 2024. (p. 31) ↗
  283. N. Nanda. Induction mosaic. Neel Nanda Blog, 2022a. URL https://neelnanda.io/mosaic. (p. 21) ↗
  284. N. Nanda. Neuroscope: A website for mechanistic interpretability of language models. Website, 2022b. URL https://neuroscope.io/. (p. 19) ↗
  285. N. Nanda. Attribution patching: Activation patching at industrial scale. https://www.neelnanda.io/mecha nistic-interpretability/attribution-patching, 2023. (p. 12) ↗
  286. N. Nanda and J. Bloom. Transformerlens. Github Repository, 2022. URL https://github.com/neelnanda -io/TransformerLens. (p. 30) ↗
  287. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=9XFSbDPmdW. (p. 27) ↗
  288. N. Nanda, A. Lee, and M. Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp…
  289. N. Nanda, S. Rajamanoharan, J. Kramár, and R. Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. AI Alignment Forum, 2023c. URL https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall. (p. 20, 29) ↗
  290. C. Neo, S. B. Cohen, and F. Barez. Interpreting context look-ups in transformers: Investigating attention-mlp interactions, 2024. URL https://arxiv.org/abs/2402.15055. (p. 26) ↗
  291. A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pp. 3395–3403, Red Hook, NY, USA, 2016.… find
  292. R. Nogueira, Z. Jiang, and J. Lin. Investigating the limitations of transformers with simple arithmetic tasks, 2021. (p. 31) nostalgebraist. Interpreting GPT: the logit lens.
  293. AI Alignment Forum, 2020. URL https://www.alignmentf orum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. (p. 17) ↗
  294. B.-D. Oh and W. Schuler. Token-wise decomposition of autoregressive language model hidden states for analyzing model predictions. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.… find
  295. C. Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/2022/mech-interp-essay. (p. 11, 13) ↗
  296. C. Olah. Distributed representations: Composition & superposition. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/superposition-composition/index.html. (p. 15) ↗
  297. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. An overview of early vision in inceptionv1. Distill, 2020a. doi: 10.23915/distill.00024.002. https://distill.pub/2020/circuits/early-vision. (p. 23) — [doi:10.23915/distill.00024.002](https://doi.org/10.23915/distill.00024.002) ↗
  298. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 2020b. doi: 10.23915/distill.00024.001. URL https://distill.pub/2020/circuits/zoom-in. (p. 15, 23) — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  299. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311–3325, 1997. ISSN 0042-6989. doi: https://doi.org/10.1016/S0042-6989(97 )00169-7. URL https://www.sciencedirect.com/science/article/pii/S0042698997001697. (p. 15) — [doi:10.1016/s0042-6989(97](https://doi.org/10.1016/s0042-6989%2897) ↗
  300. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah.… — In-context Learning and Induction Heads
  301. F. Ortu, Z. Jin, D. Doimo, M. Sachan, A. Cazzaniga, and B. Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. Computing Research Repository, arXiv:2402.11655, 2024. URL https://arxiv.org/abs/2402.11655. (p. 29) ↗
  302. G. PAIR Team. Saliency: Framework-agnostic implementation for state-of-the-art saliency methods, 2023. URL https://github.com/PAIR-code/saliency. (p. 30) ↗
  303. K. Pal, J. Sun, A. Yuan, B. Wallace, and D. Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In J. Jiang, D. Reitter, and S. Deng (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp. 548–560, Singapore, December 2023. Associat…
  304. L. Parcalabescu and A. Frank. On measuring faithfulness or self-consistency of natural language explanations, 2023. (p. 19) find
  305. K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. Arxiv, 2023a. URL https://arxiv.org/abs/2311.03658. (p. 14) ↗
  306. S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. M adry. Trak: attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023b. (p. 8) find
  307. G. Paulo, T. Marshall, and N. Belrose. Does transformer interpretability transfer to rnns?, 2024. (p. 15) find
  308. J. Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, pp. 411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001. (p. 12) find
  309. J. Pearl. Causality. Cambridge University Press, 2 edition, 2009. doi: 10.1017/CBO9780511803161. (p. 9) find
  310. M. E. Peters, M. Neumann, L. Zettlemoyer, and W.-t. Yih. Dissecting contextual word embeddings: Architecture and representation. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499–1509, Bruss…
  311. P. Pezeshkpour, S. Jain, S. Singh, and B. Wallace. Combining feature and instance attribution to detect artifacts. In S. Muresan, P. Nakov, and A. Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 1934–1946, Dublin, Ireland, May 2022. Association for Comp… find
  312. C. Pierse. Transformers Interpret, February 2021. URL https://github.com/cdpierse/transformers-i nterpret. (p. 30) ↗
  313. T. Pimentel, J. Valvoda, R. H. Maudslay, R. Zmigrod, A. Williams, and R. Cotterell. Information-theoretic probing for linguistic structure. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609…
  314. A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. (p. 27)
  315. N. Prakash, T. R. Shaham, T. Haklay, Y. Belinkov, and D. Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=8sKcAWOf2D. (p. 27) ↗
  316. G. Puccetti, A. Rogers, A. Drozd, and F. Dell’Orletta. Outlier dimensions that disrupt transformers are driven by frequency. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1286–1304, Abu Dhabi, United Arab Emirates, Decemb… find
  317. J. Qi, R. Fernández, and A. Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10650–10666, Singapore, December 2023. Associat…
  318. L. Quirke, L. Heindrich, W. Gurnee, and N. Nanda. Training dynamics of contextual n-grams in language models, 2023. URL https://arxiv.org/abs/2311.00863. (p. 23) ↗
  319. A. Radford, R. Jozefowicz, and I. Sutskever. Learning to generate reviews and discovering sentiment. Arxiv, 2017. URL https://arxiv.org/abs/1704.01444. (p. 14) ↗
  320. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. OpenAI Blog, 2018. URL https://openai.com/research/language-unsupervised. (p. 2) ↗
  321. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019. URL https://d4mucfpksywv.cloudfront.net/better-langu age-models/language_models_are_unsupervised_multitask_learners.pdf. (p. 2, 21) ↗
  322. A. Raganato and J. Tiedemann. An analysis of encoder representations in transformer-based machine translation. In T. Linzen, G. Chrupała, and A. Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 287–297, Brussels, Belgium, No…
  323. S. Rajamanoharan. Progress update 1 from the gdm mech interp team. improving ghost grads. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/progress-updat e-1-from-the-gdm-mech-interp-team-full-update. (p. 57) ↗
  324. S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, and N. Nanda. Improving dictionary learning with gated sparse autoencoders. ArXiv, 2024a. (p. 16, 17, 57)
  325. S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. ArXiv, 2024b. URL https://arxiv.org/abs/2407.14435. (p. 17) — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) ↗
  326. S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237…
  327. S. Ravfogel, M. Twiton, Y. Goldberg, and R. D. Cotterell. Linear adversarial concept erasure. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Researc… find
  328. M. T. Ribeiro, S. Singh, and C. Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1135–1144, 2016. (p. 7)
  329. P. M. Riechers. Geometry and dynamics of layernorm. arXiv, 2024. URL https://arxiv.org/abs/2405.0 4134. (p. 3) ↗
  330. A. Rogers, O. Kovaleva, and A. Rumshisky. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8:842–866, 01 2021. ISSN 2307-387X. doi: 10.1162/tacl_a_00349. URL https://doi.org/10.1162/tacl_a_00349. (p. 2, 14) ↗
  331. W. Rudman, C. Chen, and C. Eickhoff. Outlier dimensions encode task specific knowledge. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14596–14605, Singapore, December 2023. Association for Computational Linguis… find
  332. C. Rushing and N. Nanda. Explorations of self-repair in language models, 2024. URL https://arxiv.org/abs/2402.15390. (p. 12) — [arXiv:2402.15390](https://arxiv.org/abs/2402.15390) ↗
  333. T. Räuker, A. Ho, S. Casper, and D. Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. Arxiv, 2023. URL https://arxiv.org/abs/2207.13243. (p. 2) — [arXiv:2207.13243](https://arxiv.org/abs/2207.13243) ↗
  334. M. Sakarvadia, A. Khan, A. Ajith, D. Grzenda, N. Hudson, A. Bauer, K. Chard, and I. Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism, 2023. (p. 18) find
  335. S. Sanyal and X. Ren. Discretized integrated gradients for explaining language models. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10285–10299, Online and Punta Cana, Dominican Republic, Nov… find
  336. G. Sarti, N. Feldhus, L. Sickert, O. van der Wal, M. Nissim, and A. Bisazza. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 421–435, Toronto, Canada,…
  337. G. Sarti, G. Chrupała, M. Nissim, and A. Bisazza. Quantifying the plausibility of context reliance in neural machine translation. In The Twelfth International Conference on Learning Representations (ICLR 2024), Vienna, Austria, May 2024. OpenReview. URL https://openreview.net/forum?id=XTHfNGI3zT. (p… ↗
  338. H. Shah, A. Ilyas, and A. Madry. Decomposing and editing predictions by modeling model computation. ArXiv, 2024. URL https://arxiv.org/abs/2404.11534. (p. 10) — [arXiv:2404.11534](https://arxiv.org/abs/2404.11534) ↗
  339. T. R. Shaham, S. Schwettmann, F. Wang, A. Rajaram, E. Hernandez, J. Andreas, and A. Torralba. A multimodal automated interpretability agent. Arxiv, 2024. URL https://arxiv.org/abs/2404.14394. (p. 31) — [arXiv:2404.14394](https://arxiv.org/abs/2404.14394) ↗
  340. L. S. Shapley. A value for n-person games. In H. W. Kuhn and A. W. Tucker (eds.), Contributions to the Theory of Games II, pp. 307–317. Princeton University Press, Princeton, 1953. (p. 7) find
  341. L. Sharkey, D. Braun, and B. Millidge. Taking features out of superposition with sparse autoencoders. AI Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/inter im-research-report-taking-features-out-of-superposition. (p. 16) ↗
  342. A. S. Sharma, D. Atkinson, and D. Bau. Locating and editing factual associations in mamba, 2024a. (p. 28)
  343. P. Sharma, J. T. Ash, and D. Misra. The truth is in there: Improving reasoning with layer-selective rank reduction. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=ozX92bu8VA. (p. 18, 29) ↗
  344. N. Shazeer. Glu variants improve transformer. ArXiv, 2020. (p. 16) find
  345. A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 3145–3153. JMLR.org, 2017. (p. 7)
  346. A. Shrikumar, J. Su, and A. Kundaje. Computationally efficient measures of internal neuron importance. ArXiv, abs/1807.09946, 2018. URL https://api.semanticscholar.org/CorpusID:50787065. (p. 12) — [arXiv:1807.09946](https://arxiv.org/abs/1807.09946) ↗
  347. N. Y. Siegel, O.-M. Camburu, N. Heess, and M. Perez-Ortiz. The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models, 2024. (p. 32) find
  348. K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In The Second International Conference on Learning Representations, 2014. URL http://arxiv.org/abs/1312.6034. (p. 6, 19) ↗
  349. A. K. Singh, T. Moskovitz, F. Hill, S. C. Y. Chan, and A. M. Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation, 2024a. (p. 21) find
  350. C. Singh, J. P. Inala, M. Galley, R. Caruana, and J. Gao. Rethinking interpretability in the era of large language models. ArXiv, abs/2402.01761, 2024b. URL https://api.semanticscholar.org/CorpusID:26741253 0. (p. 32) ↗
  351. S. Singh, S. Ravfogel, J. Herzig, R. Aharoni, R. Cotterell, and P. Kumaraguru. Mimic: Minimally modified counterfactuals in the representation space. Arxiv, 2024c. URL https://arxiv.org/abs/2402.09631. (p. 14, 15) — [arXiv:2402.09631](https://arxiv.org/abs/2402.09631) ↗
  352. L. Sixt, M. Granz, and T. Landgraf. When explanations lie: Why many modified BP attributions fail. In H. D. III and A. Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9046–9057. PMLR, 13–18 Jul 2020. URL… find
  353. D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise, 2017. (p. 7) find
  354. P. Smolensky. Neural and conceptual interpretation of PDP models, pp. 390–431. MIT Press, Cambridge, MA, USA, 1986. ISBN 0262631105. (p. 15) find
  355. N. Stoehr, M. Gordon, C. Zhang, and O. Lewis. Localizing paragraph memorization in language models, 2024. URL https://arxiv.org/abs/2403.19851. (p. 29) — [arXiv:2403.19851](https://arxiv.org/abs/2403.19851) ↗
  356. A. Stolfo, Y. Belinkov, and M. Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035–7052, Singapore… find
  357. A. Stolfo, Y. Belinkov, and M. Sachan. Understanding arithmetic reasoning in language models using causal mediation analysis. Arxiv, 2023b. URL https://arxiv.org/abs/2305.15054. (p. 11, 26) ↗
  358. A. Stolfo, B. Wu, W. Gurnee, Y. Belinkov, X. Song, M. Sachan, and N. Nanda. Confidence regulation neurons in language models. ArXiv, 2024. URL https://arxiv.org/abs/2406.16254. (p. 23, 24) — [arXiv:2406.16254](https://arxiv.org/abs/2406.16254) ↗
  359. X. Suau, L. Zappella, and N. Apostoloff. Finding experts in transformer models, 2020. (p. 23) find
  360. X. Suau, L. Zappella, and N. Apostoloff. Self-conditioning pre-trained language models. International Conference on Machine Learning, 2022. URL https://proceedings.mlr.press/v162/cuadros22a/cuad ros22a.pdf. (p. 23) ↗
  361. M. Sun, X. Chen, J. Z. Kolter, and Z. Liu. Massive activations in large language models, 2024. URL https://arxiv.org/abs/2402.17762. (p. 22) ↗
  362. M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3319–3328. JMLR.org, 2017. (p. 7) find
  363. A. Syed, C. Rager, and A. Conmy. Attribution patching outperforms automated circuit discovery. Arxiv, 2023. URL https://arxiv.org/abs/2310.10348. (p. 12) ↗
  364. S. Takase, S. Kiyono, S. Kobayashi, and J. Suzuki. B2T connection: Serving stability and performance in deep transformers. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3078–3095, Toronto, Canada, July 2023. Association… find
  365. T. Tang, W. Luo, H. Huang, D. Zhang, X. Wang, X. Zhao, F. Wei, and J.-R. Wen. Language-specific neurons: The key to multilingual capabilities in large language models, 2024. URL https://arxiv.org/abs/2402 .16438. (p. 23) ↗
  366. D. A. Tarzanagh, Y. Li, C. Thrampoulidis, and S. Oymak. Transformers as support vector machines, 2024. (p. 31) find
  367. G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor… find
  368. A. Templeton, T. Conerly, J. Marcus, T. Henighan, A. Golubeva, and T. Bricken. Circuits updates - february 2024. update on dictionary learning improvements. find
  369. Transformer Circuits Thread, 2024a. URL https://transformer-circuits.pub/2024/feb-update/index.html. (p. 57) ↗
  370. A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and T. Henighan. Scaling monosemanticity: Extract… find
  371. I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In A. Korhonen, D. Traum, and L. Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601, Florence, Italy, July 2019a. Association for Computational Linguist…
  372. I. Tenney, P. Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. V. Durme, S. Bowman, D. Das, and E. Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019b. URL https://open…
  373. I. Tenney, J. Wexler, J. Bastings, T. Bolukbasi, A. Coenen, S. Gehrmann, E. Jiang, M. Pushkarna, C. Radebaugh, E. Reif, and A. Yuan. The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models. In Q. Liu and D. Schlangen (eds.), Proceedings of the 2020 Conf… find
  374. I. Tenney, R. Mullins, B. Du, S. Pandya, M. Kahng, and L. Dixon. Interactive prompt debugging with sequence salience. Arxiv, 2024. URL https://arxiv.org/abs/2404.07498. (p. 30) — [arXiv:2404.07498](https://arxiv.org/abs/2404.07498) ↗
  375. Y. Tian, Y. Wang, B. Chen, and S. Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer, 2023. (p. 31) find
  376. Y. Tian, Y. Wang, Z. Zhang, B. Chen, and S. S. Du. JoMA: Demystifying multilayer transformers via joint dynamics of MLP and attention. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=LbJqRGNYCf. (p. 31) ↗
  377. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996. doi: https://doi.org/10.1111/j.2517-6161.1996.tb02080.x. URL https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1996.tb02080.x. (p.… ↗
  378. C. Tigges, O. J. Hollinsworth, A. Geiger, and N. Nanda. Linear representations of sentiment in large language models. Arxiv, 2023. URL https://arxiv.org/abs/2310.15154. (p. 15, 25) ↗
  379. W. Timkey and M. van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4527–4546,…
  380. E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. LLMs represent contextual tasks as compact function vectors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwyxtyMwaG. (p. 25, 26) ↗
  381. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  382. I. Tufanov, K. Hambardzumyan, J. Ferrando, and E. Voita. Lm transparency tool: Interactive tool for analyzing transformer language models. Arxiv, 2024. URL https://arxiv.org/abs/2404.07004. (p. 31) ↗
  383. A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization, 2023. (p. 15) find
  384. M. Turpin, J. Michael, E. Perez, and S. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. ArXiv, abs/2305.04388, 2023. URL https://api.semanticscholar.org/CorpusID:258556812. (p. 19) ↗
  385. A. Variengien. Some common confusion about induction heads. LessWrong, 2023. URL https://www.less wrong.com/posts/nJqftacoQGKurJ6fv/some-common-confusion-about-induction-heads. (p. 21) ↗
  386. A. Variengien and E. Winsor. Look before you leap: A universal emergent decomposition of retrieval tasks in language models, 2023. URL https://arxiv.org/abs/2312.10091. (p. 26, 29) — [arXiv:2312.10091](https://arxiv.org/abs/2312.10091) ↗
  387. V. Varma, R. Shah, Z. Kenton, J. Kram’ar, and R. Kumar. Explaining grokking through circuit efficiency. ArXiv, abs/2309.02390, 2023. URL https://api.semanticscholar.org/CorpusID:261557247. (p. 27) — [arXiv:2309.02390](https://arxiv.org/abs/2309.02390) ↗
  388. N. Varshney, W. Yao, H. Zhang, J. Chen, and D. Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation, 2023. URL https://arxiv.org/abs/23 07.03987. (p. 27) ↗
  389. H. Vasconcelos, M. Jörke, M. Grunde-McLaughlin, T. Gerstenberg, M. S. Bernstein, and R. Krishna. Explanations can reduce overreliance on ai systems during decision-making. Proc. ACM Hum.-Comput. Interact., 7 (CSCW1), apr 2023. doi: 10.1145/3579605. URL https://doi.org/10.1145/3579605. (p. 32) ↗
  390. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curr…
  391. A. Veit, M. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pp. 550–558, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. (p. 5) find
  392. J. Vig. A multiscale visualization of attention in the transformer model. In M. R. Costa-jussà and E. Alfonseca (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37–42, Florence, Italy, July 2019. Association for Computational…
  393. J. Vig and Y. Belinkov. Analyzing the structure of attention in a transformer language model. In T. Linzen, G. Chrupała, Y. Belinkov, and D. Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63–76, Florence, Italy, August 2019. A…
  394. J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388… find
  395. E. Voita and I. Titov. Information-theoretic probing with minimum description length. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183–196, Online, November 2020. Association for Computational Lin…
  396. E. Voita, R. Sennrich, and I. Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In K. Inui, J. Jiang, V. Ng, and X. Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processin…
  397. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In A. Korhonen, D. Traum, and L. Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p…
  398. E. Voita, R. Sennrich, and I. Titov. Analyzing the source and target contributions to predictions in neural machine translation. In C. Zong, F. Xia, W. Li, and R. Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  399. E. Voita, J. Ferrando, and C. Nalmpantis. Neurons in large language models: Dead, n-gram, positional. Arxiv, 2023. URL https://arxiv.org/abs/2309.04827. (p. 4, 14, 23, 25) ↗
  400. J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (eds.), Proceedings of the 40th International Conference on Machi…
  401. K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=NpsVSN6o4ul. (p. 9, 10, 11… ↗
  402. Q. Wang, T. Anikina, N. Feldhus, J. van Genabith, L. Hennig, and S. Möller. Llmcheckup: Conversational examination of large language models via interpretability tools, 2024. (p. 32) find
  403. S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li. Knowledge editing for large language models: A survey. ArXiv, abs/2310.16218, 2023b. URL https://api.semanticscholar.org/CorpusID:264487359. (p. 30) ↗
  404. X. Wang, K. Wen, Z. Zhang, L. Hou, Z. Liu, and J. Li. Finding skill neurons in pre-trained transformer-based language models. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132–11152, Abu Dhabi, United A…
  405. D. Wei, R. Nair, A. Dhurandhar, K. R. Varshney, E. Daly, and M. Singh. On the safety of interpretable machine learning: A maximum deviation approach. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9866–98… find
  406. G. Weiss, Y. Goldberg, and E. Yahav. Thinking like transformers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11080–11090. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v13… ↗
  407. K. Wen, Y. Li, B. Liu, and A. Risteski. Transformers are uninterpretable with myopic methods: a case study with bounded dyck grammars. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 38723–38766. Curran… find
  408. N. Wichers, C. Denison, and A. Beirami. Gradient-based language model red teaming, 2024. (p. 19) find
  409. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In…
  410. B. Wright and L. Sharkey. Addressing feature suppression in saes. AI ALIGNMENT FORUM, 2024. URL https://www.alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppression -in-saes. (p. 16) ↗
  411. W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu. Retrieval head mechanistically explains long-context factuality. Arxiv, 2024a. URL https://arxiv.org/abs/2404.15574. (p. 29) ↗
  412. Z. Wu, K. D’Oosterlinck, A. Geiger, A. Zur, and C. Potts. Causal proxy models for concept-based model explanations. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023a. (p. 13) find
  413. Z. Wu, A. Geiger, T. Icard, C. Potts, and N. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 78205–78226. Curran Associates, I…
  414. Z. Wu, A. Arora, Z. Wang, A. Geiger, D. Jurafsky, C. D. Manning, and C. Potts. Reft: Representation finetuning for language models, 2024b. (p. 13, 15, 31, 32)
  415. Z. Wu, A. Geiger, A. Arora, J. Huang, Z. Wang, N. D. Goodman, C. D. Manning, and C. Potts. pyvene: A library for understanding and improving pytorch models via interventions, 2024c. (p. 30) find
  416. Z. Wu, A. Geiger, J. Huang, A. Arora, T. Icard, C. Potts, and N. D. Goodman. A reply to makelov et al. (2023)’s "interpretability illusion" arguments, 2024d. (p. 12) find
  417. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. Arxiv, 2023. URL https://arxiv.org/abs/2309.17453. (p. 22) ↗
  418. S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=RdJVFCHjUMI. (p. 31) ↗
  419. R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning. JMLR.org, 2020. (p. 3) find
  420. S. Yang, S. Huang, W. Zou, J. Zhang, X. Dai, and J. Chen. Local interpretation of transformer based on linear decomposition. In A. Rogers, J. Boyd-Graber, and N. Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10270… find
  421. Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, and N. Zhang. Editing large language models: Problems, methods, and opportunities. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10222–10240, Singapo… find
  422. K. Yin and G. Neubig. Interpreting language models with contrastive explanations. In Y. Goldberg, Z. Kozareva, and Y. Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 184–198, Abu Dhabi, United Arab Emirates, December 2022. Association for Com… find
  423. Q. Yu, J. Merullo, and E. Pavlick. Characterizing mechanisms for factual recall in language models. In H. Bouamor, J. Pino, and K. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9924–9959, Singapore, December 2023a. Association for Computatio… find
  424. Y. Yu, S. Buchanan, D. Pai, T. Chu, Z. Wu, S. Tong, B. D. Haeffele, and Y. Ma. White-box transformers via sparse rate reduction. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https://openreview.net/forum?id=THfl8hdVxH. (p. 31) ↗
  425. Z. Yu and S. Ananiadou. Locating factual knowledge in large language models: Exploring the residual stream and analyzing subvalues in vocabulary space, 2024. URL https://arxiv.org/abs/2312.12141. (p. 26) — [arXiv:2312.12141](https://arxiv.org/abs/2312.12141) ↗
  426. M. Yuksekgonul, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/foru… ↗
  427. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (eds.), Computer Vision – ECCV 2014, pp. 818–833, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10590-1. (p. 19)
  428. B. Zhang and R. Sennrich. Root mean square layer normalization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc. (p. 3) find
  429. F. Zhang and N. Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Hf17y6u9BC. (p. 9, 10) ↗
  430. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. (p. 23)
  431. Z. Zhao and B. Shan. Reagent: A model-agnostic feature attribution method for generative language models, 2024. (p. 7) find
  432. C. Zheng, F. Yin, H. Zhou, F. Meng, J. Zhou, K.-W. Chang, M. Huang, and N. Peng. On prompt-driven safeguarding for large language models. ArXiv, 2024. URL https://arxiv.org/abs/2402.03485. (p. 15) ↗
  433. Z. Zhong, Z. Liu, M. Tegmark, and J. Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27223–27250. Curran Assoc… find
  434. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. In International Conference on Learning Representations (ICLR), 2015. (p. 19) find
  435. H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. M. Susskind, S. Bengio, and P. Nakkiran. What algorithms can transformers learn? a study in length generalization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AssIuHnmHX. (p. 3… ↗
  436. A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks. Representation engineering: A top-down approach to ai transparency. Arxiv…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ferrando-2024-primer-inner-workings-transformer-based/