Elamur Research⌕
☰
home / papers / 2023 / Copy Suppression: Comprehensively Understanding an Attention Head

Copy Suppression: Comprehensively Understanding an Attention Head

Paper · 2023 · Callum McDougall, Arthur Conmy, Cody Rushing et al.

Summary

The paper addresses a long-standing difficulty in mechanistic interpretability. Explanations of model components usually hold only on narrow tasks, and "Negative Heads" that push against the right answer have puzzled earlier studies. The authors fully reverse-engineer one such head, L10H7 in GPT-2 Small, across its whole training distribution, using OpenWebText. Their account is called copy suppression. Earlier layers predict a token that already appeared in the context. The head detects this, attends back to that earlier occurrence, and lowers its logit, which improves calibration. The evidence comes from several directions: behavioral sampling, path-based ablations, weights-based analysis of the OV and QK circuits (using MLP0(W_E) as an effective embedding), and a custom ablation called CSPA. CSPA keeps only the copy-suppression mechanism and still retains most of the head's effect. The work also links copy suppression to self-repair, where downstream parts compensate after a component is ablated. This matters because negative and backup components distort ablation-based importance estimates.

Key points

  • Among high-impact OpenWebText completions (the top 5%, which carry half of the head's loss reduction), about 80% matched the three-step copy-suppression criteria.
  • Most of L10H7's effect on loss runs through its direct path to the final logits.
  • Weights-based circuit analysis supports the mechanism:
    • OV circuit: for 84.70% of vocabulary tokens, the head's own token falls among the 10 most suppressed outputs. For 98.86% of tokens, it falls in the bottom 5%.
    • QK circuit: the head attends to the currently predicted token for 95.72% of the vocabulary.
  • CSPA, the copy suppression-preserving ablation, retains 76.9% of L10H7's behavior, measured by KL divergence.
  • In the IOI task, copy suppression accounts for 39% of self-repair. Interactive demos are available at copy-suppression.streamlit.app.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States126 works · 58,144 citations🏢Google DeepMindLondon · United Kingdom82 works · 84,829 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath and Neel Nanda. 2023. Copy Suppression: Comprehensively Understanding an Attention Head. arXiv:2310.04625.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.04625 · arXiv 2310.04625

@misc{mcdougall2023copy,
  title = {{Copy Suppression: Comprehensively Understanding an Attention Head}},
  author = {Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath and Neel Nanda},
  year = {2023},
  doi = {10.48550/arxiv.2310.04625},
  eprint = {2310.04625},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.04625}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017. 2 find
  2. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. 9
  3. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. 2, 9 ↗
  4. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021. 2 — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) find
  5. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. 2020. doi: 10.23915/distill.00024. https://distill.pub/2020/circuits. 2 — [doi:10.23915/distill.00024](https://doi.org/10.23915/distill.00024) ↗
  6. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 4(3):e15, 2019. 2 find
  7. Lawrence Chan, Adria Garriga-Alonso, Nix Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: A method for rigorously testing interpretability hypotheses. Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/JvZhhzycH… ↗
  8. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023. 2 find
  9. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02535, 2022. 9
  10. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  11. Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021. 2 find
  12. Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019. URL https://Skylion007.github.io/OpenWebTextCorpus. 3 ↗
  13. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023. 3
  14. Klaus Greff, Rupesh K. Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation, 2017. 9 find
  15. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023. 2, 9
  16. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations, 2023. 9 find
  17. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. 2, 17
  18. Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer, 2023. URL https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/ a-circuit-for-python-docstrings-in-a-4-layer-attention-only. 2 ↗
  19. Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. Uncertainty in natural language processing: Sources, quantification, and applications, 2023. 15 find
  20. Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?, 2020. 2
  21. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations, 2023. 7, 28
  22. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. CoRR, abs/2006.14032, 2020. URL https://arxiv.org/abs/2006.14032. 2 — [arXiv:2006.14032](https://arxiv.org/abs/2006.14032) ↗
  23. Neel Nanda and Joseph Bloom. Transformerlens, 2022. neelnanda-io/TransformerLens. 17 find
  24. URL https://github.com/nostalgebraist. interpreting gpt: the logit lens, 2020. URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. 3, 9, 28 ↗
  25. Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. https://www.transformer-circuits.pub/2022/mech-interp-essay, 2022. 2 ↗
  26. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads, 2022. URL https://transformer-circuits.pub/2022/ in-context-learning-and-induction-heads/index.html. 2, 12, 27, 2… ↗
  27. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment, 2017. 2
  28. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks, 2023. 2 find
  29. Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37–42. Association for Computational Linguistics, 2019. doi: 10.18653/v1/P19-3007. URL https://aclanthol…
  30. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned, 2019. 2, 9
  31. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional, 2023. 9 find
  32. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6… ↗
  33. Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca, 2023. 2

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mcdougall-2023-copy-suppression-comprehensively-understanding-attention/