Elamur Research⌕
☰
home / papers / 2023 / Successor Heads: Recurring, Interpretable Attention Heads In The Wild

Successor Heads: Recurring, Interpretable Attention Heads In The Wild

Paper · 2023 · Rhys Gould, Euan Ong, George Ogden et al.

Summary

The paper tackles a long-standing gap in mechanistic interpretability: few model components have been shown to be both understandable and present across many real LLMs, not just toy models. The authors identify "successor heads," attention heads that map a token from an ordered sequence (a number, month, weekday, letter) to the next item, such as Monday to Tuesday. They build a succession dataset spanning eight token families and score each head through an effective OV circuit that passes embeddings through MLP0 before the head. A head qualifies if it ranks the correct successor highest for over half of the tokens. Such heads appear in Pythia, GPT-2 and Llama-2, across model sizes from 31M to 12B parameters. A case study of Pythia-1.4B shows that token representations split into an index part and a domain part. Sparse autoencoders then reveal shared "mod-10" features, which the authors use to steer the head's outputs. The heads also help lower loss on succession examples in training data. Together, these results offer evidence for a weak form of universality.

Key points

  • The succession dataset includes eight tasks: Numbers (1–200) and Letters, two spelled-out tasks (Number words and Cardinal words), two weekday tasks (Days and Day prefixes) and two month tasks (Months and Month prefixes). It contains 1041 tokens in total, including tokenizer variants.
  • The strongest successor head in Pythia-1.4B is L12H0. The authors also examine the best heads in GPT-2 XL and Llama-2 7B.
  • Mod-10 features (f0–f9) are shared across domains. For example, f2 is active for '2', '32', 'February', 'second' and 'twelve'. Vector arithmetic on these features changes which successor the head predicts.
  • When projected index and domain parts are recombined and passed through the head, top-1 accuracy is 0.90 on held-out token pairs. It falls to 0.125 on out-of-distribution Roman numerals.
  • The authors trained linear probes for each modulus from 2 to 25:
    • The mod-10 probe has the best out-of-distribution accuracy and recovers the index of 10/12 months and 4/7 days.
    • The mod-7 probe gets no weekdays right.
    • The mod-12 probe gets only 8/12 months right.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 156 works

Cite

Cite as Rhys Gould, Euan Ong, George Ogden and Arthur Conmy. 2023. Successor Heads: Recurring, Interpretable Attention Heads In The Wild. arXiv:2312.09230.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.09230 · arXiv 2312.09230

@misc{gould2023successor,
  title = {{Successor Heads: Recurring, Interpretable Attention Heads In The Wild}},
  author = {Rhys Gould, Euan Ong, George Ogden and Arthur Conmy},
  year = {2023},
  doi = {10.48550/arxiv.2312.09230},
  eprint = {2312.09230},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.09230}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017. 10 find
  2. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. 18
  3. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… find
  4. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. 2 ↗
  5. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… find
  6. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023. URL https://arxiv.org/abs/2302. 03025. 2, 10 ↗
  7. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023. 2 find
  8. Stanford CRFM. Mistral: A framework for transparent and accessible large-scale language model training. https://github.com/stanford-crfm/mistral, 2021. 19 ↗
  9. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. 5, 14 find
  10. Kevin Du, Lucas Torroba Hennigen, Niklas Stoehr, Alexander Warstadt, and Ryan Cotterell. Generalizing backpropagation for gradient-based interpretability, 2023. 11 find
  11. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  12. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. 1, 2, 6, 10 find
  13. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023. 11
  14. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023. 10
  15. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. 2, 11
  16. Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer, 2023. URL https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/ a-circuit-for-python-docstrings-in-a-4-layer-attention-only. 2 ↗
  17. Mikhail L. One is 1: Analyzing activations of numerical words vs digits 1, Jul 2023. 11 find
  18. Michael Lan and Fazl Barez. Locating cross-task sequence continuation circuits in transformers, 2023. 11 find
  19. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations?, 2016. 2 find
  20. Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. arXiv preprint, 2023. 3
  21. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations, 2023. 21
  22. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic, 2023. 11 find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality, 2013. 6, 11
  24. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW. 2, 10 ↗
  25. Andrew Ng. Sparse autoencoder. Online Course Notes, Stanford University, 2011. URL https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf. 5 ↗
  26. Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. https://www.transformer-circuits.pub/2022/mech-interp-essay, 2022. 1, 10 ↗
  27. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Featture visualization. Distill, 2017. doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization. 10 — [doi:10.23915/distill.00007](https://doi.org/10.23915/distill.00007) ↗
  28. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. 2, 10 — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) find
  29. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads, 2022. URL https://transformer-circuits.pub/2022/ in-context-learning-and-induction-heads/index.html. 2, 10, 11 ↗
  30. OpenAI. Gpt-4 technical report, 2023. 2
  31. Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009. ISBN 978-0-521-89560-6. doi: 10.1017/CBO9780511803161. 21 find
  32. Lee Sharkey, Dan Braun, and Beren Millidge. [interim research report] taking features out of superposition with sparse autoencoders, 2022. URL https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/ interim-research-report-taking-features-out-of-superposition. 14 ↗
  33. Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models, 2022. 11
  34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  35. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, 2023. 11 find
  36. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6… ↗
  37. Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca, 2023. 2

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gould-2023-successor-heads-recurring-interpretable-attention/