Elamur Research⌕
☰
home / papers / 2023 / Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

Paper · 2023 · Tom Lieberum, Matthew Rahtz, Janos Kramar et al.

Summary

Circuit analysis has mostly been demonstrated on small networks, such as two-layer attention-only transformers or the 117M GPT-2. It has been unclear whether the method works on frontier-scale systems. This Google DeepMind study tests that question on the 70B-parameter Chinchilla. It asks how the model outputs the right answer letter (A–D) on MMLU multiple-choice questions once it knows which answer text is correct, a symbol-manipulation skill that smaller Chinchilla variants lack. The authors combine logit attribution, attention-pattern inspection and activation patching to find a small set of output components (attention heads and MLPs). Chief among them are the so-called "correct letter" heads, whose attention lands on the right answer's label and which then copy that label into the logits. Restricting these heads' queries, keys and values to low-dimensional SVD subspaces keeps MMLU performance intact. The query/key directions partly encode an ordinal, position-in-list feature, while the values carry token identity. This account breaks down partially when the labels are randomized letters. The study shows that standard tools scale to large models, but the features they reveal can be semantically messy.

Key points

  • The model studied is Chinchilla 70B, with 80 layers, 64 heads per layer, RMSNorm and relative positional embeddings. The analysis uses 0-shot prompts from a 6-topic subset of MMLU, a benchmark of roughly 16,000 questions.
  • On 5-shot MMLU, accuracy scored by label is 25% for the 1B model, 26% for 7B and 68% for 70B. Scored by answer text, it is 27%, 32% and 65%. This suggests the 7B model has some of the knowledge but lacks the letter-selection ability.
  • Only the 70B model beats chance on a synthetic multiple-choice task that requires no factual knowledge.
  • Correct-letter heads compress to 3-dimensional query, key and value subspaces with no loss on normal MMLU prompts. The query/key subspaces approximately capture an "Nth item in an enumeration" feature.
  • The identified components form groups that appear to act independently, some generalizing and some not. The authors call the overall results noisy and sometimes contradictory, and they argue for better and more automated interpretability tools.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 58 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Scaling laws · 30 works

Cite

Cite as Tom Lieberum, Matthew Rahtz, Janos Kramar, Neel Nanda, Geoffrey Irving, Rohin Shah and Vladimir Mikulik. 2023. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla. arXiv:2307.09458.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.09458 · arXiv 2307.09458

@misc{lieberum2023does,
  title = {{Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla}},
  author = {Tom Lieberum, Matthew Rahtz, Janos Kramar, Neel Nanda, Geoffrey Irving, Rohin Shah and Vladimir Mikulik},
  year = {2023},
  doi = {10.48550/arxiv.2307.09458},
  eprint = {2307.09458},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.09458}
}

References

53 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.
  2. N. Belrose, Z. Furman, L. Smith, D. Halawi, L. McKinney, I. Ostrovsky, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.
  3. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html , 2023. ↗
  4. J. I. Bloom and P. Colognese. Decision transformer interpretability. https://www.alignmentforum.org/posts/bBuBDJBYHt39Q5zZy/decision-transformer-interpretability , 2023. ↗
  5. N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/distill.00024. https://distill.pub/2020/circuits. — [doi:10.23915/distill.00024](https://doi.org/10.23915/distill.00024) ↗
  6. S. Cao, V. Sanh, and A. M. Rush. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.03514, 2021. find
  7. L. Chan, A. Garriga-Alonso, N. Goldowsky-Dill, R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses. https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/ causal-scrubbing , 2022. ↗
  8. A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304.14997, 2023. find
  9. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. find
  10. G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02535, 2022.
  11. A. Y. Din, T. Karidi, L. Choshen, and M. Geva. Jump to conclusions: Short-cutting transformers with linear transformations. arXiv preprint arXiv:2303.09435, 2023. find
  12. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematic… find
  13. N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. Das-Sarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E… find
  14. N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022b. https://transformer-circuits.pub/2022/toy… ↗
  15. A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. find
  16. A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.04709, 2023a. find
  17. A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint arXiv:2303.02536, 2023b. find
  18. M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022.
  19. M. Geva, J. Bastings, K. Filippova, and A. Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023. find
  20. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022.
  21. W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023. find
  22. S. Heimersheim and J. Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer. https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/ a-circuit-for-python-docstrings-in-a-4-layer-attention-only , 2023. ↗
  23. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  24. J. Hilton, N. Cammarata, S. Carter, G. Goh, and C. Olah. Understanding RL vision. Distill, 2020. doi: 10.23915/distill.00029. https://distill.pub/2020/understanding-rl-vision. — [doi:10.23915/distill.00029](https://doi.org/10.23915/distill.00029) ↗
  25. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  26. Y.-C. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, and H. Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022. — [arXiv:2207.00112](https://arxiv.org/abs/2207.00112) find
  27. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRa: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  28. E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) find
  29. G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. find
  30. E. Jenner, A. Garriga-Alonso, and E. Zverev. A comparison of causal scrubbing, causal abstractions, and related methods. https://www.alignmentforum.org/posts/uLMWMeBG3ruoBRhMW/ a-comparison , 2023. ↗
  31. A. Jermyn, C. Olah, and T. Henighan. Circuits updates — May 2023: Attention head superposition. find
  32. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/may-update/index.html. ↗
  33. Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021. find
  34. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
  35. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  36. P. Michel, O. Levy, and G. Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
  37. N. Nanda, L. Chan, T. Liberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. nostalgebraist. interpreting GPT: the logit lens. https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens , 2… ↗
  38. C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. doi: 10.23915/distill. 00007. https://distill.pub/2017/feature-visualization. — [doi:10.23915/distill](https://doi.org/10.23915/distill) ↗
  39. C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/distill.00010. https://distill.pub/2018/building-blocks. — [doi:10.23915/distill.00010](https://doi.org/10.23915/distill.00010) ↗
  40. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah.… find
  41. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  42. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  43. J. Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669–688, 1995. ISSN 00063444. URL http://www.jstor.org/stable/2337329 . ↗
  44. J. Pearl. The do-calculus revisited. arXiv preprint arXiv:1210.4852, 2012. — [arXiv:1210.4852](https://arxiv.org/abs/1210.4852) find
  45. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022.
  46. O. Ram, L. Bezalel, A. Zicher, Y. Belinkov, J. Berant, and A. Globerson. What are you token about? dense retrieval as distributions over the vocabulary. arXiv preprint arXiv:2212.10380, 2022. find
  47. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
  48. T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023. find
  49. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
  50. K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv preprint arXiv:2211.00593, 2022. find
  51. Z. Wu, A. Geiger, C. Potts, and N. D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. arXiv preprint arXiv:2305.08809, 2023.
  52. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  53. Figure 15 \| Example of a synthetic multiple choice question. Boxes denote token boundaries. 0.5 Random 0.8 Accuracy Loss 1.0 0.0 1.0 Random 1.5 0.6 0.4 0.2 1B 7B Model Size 70B (a) Loss 0.0 1B 7B Model Size 70B (b) Accuracy Figure 16 \| Performance comparison of model sizes on the synthetic multiple… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/