Elamur Research⌕
☰
home / papers / 2023 / The Hydra Effect: Emergent Self-repair in Language Model Computations

The Hydra Effect: Emergent Self-repair in Language Model Computations

Paper · 2023 · Thomas McGrath, Matthew Rahtz, Janos Kramar et al.

Summary

The paper asks how far ablation can be trusted for working out which parts of a large language model drive its predictions. The authors study a 7-billion-parameter decoder-only model trained with the Chinchilla recipe. They prompt it with subject–relation pairs from the Counterfact dataset, so each answer requires recalling a fact. For every attention and MLP layer they compare two importance scores at the final token position. The first projects the layer's output through the model's own unembedding, in the logit lens style, and corresponds to the direct effect. The second replaces the output via resample ablation and corresponds to the total effect.

The two scores are weakly correlated, and ablation usually hurts less than the unembedding score predicts. Tracing downstream activations explains the gap. When an attention layer is removed, later attention layers increase their contribution, which the authors call the Hydra effect. At the same time, late MLPs that normally push down the top token suppress it less. This self-repair matters because circuit attribution, automated ablation pipelines and direct-logit readings of layers all assume components act independently.

Key points

  • Model and data: a Chinchilla-family 7B model trained without dropout or stochastic depth, evaluated on Counterfact prompts. Ablations use activation patches drawn from 15 alternative prompts.
  • In middle layers, where compensation is strongest, the Hydra effect and the weakened MLP suppression together restore roughly 70% of the lost logit of the predicted token. The compensation is approximately linear.
  • Late-layer "erasure" MLPs tend to lower the logit of the maximum-likelihood token, acting as a negative-feedback mechanism.
  • Layers are fairly loosely coupled: knocking out one layer changes only a few downstream layers.
  • Wang et al. (2022) observed similar "backup heads" in GPT-2 Small and attributed them to dropout. Because Chinchilla 7B used no dropout, these results refute that explanation. The authors instead speculate about a training-time "natural dropout" or superposition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik and Shane Legg. 2023. The Hydra Effect: Emergent Self-repair in Language Model Computations. arXiv:2307.15771.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.15771 · arXiv 2307.15771

@misc{mcgrath2023hydra,
  title = {{The Hydra Effect: Emergent Self-repair in Language Model Computations}},
  author = {Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik and Shane Legg},
  year = {2023},
  doi = {10.48550/arxiv.2307.15771},
  eprint = {2307.15771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15771}
}

References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. find
  2. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  3. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  4. D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597, 2018. — [arXiv:1811.10597](https://arxiv.org/abs/1811.10597) find
  5. N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.
  6. N. Cammarata, G. Goh, S. Carter, L. Schubert, M. Petrov, and C. Olah. Curve detectors. Distill, 5(6): e00024–003, 2020. find
  7. S. Carter, Z. Armstrong, L. Schubert, I. Johnson, and C. Olah. Activation atlas. Distill, 2019. doi: 10.23915/distill.00015. https://distill.pub/2019/activation-atlas. — [doi:10.23915/distill.00015](https://doi.org/10.23915/distill.00015) ↗
  8. L. Chan, A. Garriga-Alonso, N. Goldwosky-Dill, R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 2022. https://www.alignmentforum.org/posts/ JvZhhzycHu2Yd57RN/causal-scrub… ↗
  9. B. Chughtai, L. Chan, and N. Nanda. A toy model of universality: Reverse engineering how networks learn group operations. arXiv preprint arXiv:2302.03025, 2023. — [arXiv:2302.03025](https://arxiv.org/abs/2302.03025) find
  10. A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304.14997, 2023. find
  11. G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02535, 2022.
  12. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. find
  13. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematic… find
  14. N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/toy_… ↗
  15. M. Finlayson, A. Mueller, S. Gehrmann, S. Shieber, T. Linzen, and Y. Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint arXiv:2106.06087, 2021. find
  16. J. Z. Forde, C. Lovering, G. Konidaris, E. Pavlick, and M. L. Littman. Where, when & which concepts does alphazero learn? lessons from the game of hex. In AAAI Workshop on Reinforcement Learning in Games, volume 2, 2022. find
  17. A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. find
  18. A. Geiger, Z. Wu, H. Lu, J. Rozner, E. Kreiss, T. Icard, N. Goodman, and C. Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7324–7338. PMLR, 2022. find
  19. A. Geiger, C. Potts, and T. Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.04709, 2023a. find
  20. A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint arXiv:2303.02536, 2023b.
  21. M. Geva, A. Caciularu, G. Dar, P. Roit, S. Sadde, M. Shlain, B. Tamir, and Y. Goldberg. Lm-debugger: An interactive tool for inspection and intervention in transformer-based language models. arXiv preprint arXiv:2204.12130, 2022a. find
  22. M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022b.
  23. M. Glymour, J. Pearl, and N. P. Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016. find
  24. G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. ↗
  25. W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
  26. J. Y. Halpern. Actual causality. MiT Press, 2016. find
  27. J. Hilton, N. Cammarata, S. Carter, G. Goh, and C. Olah. Understanding rl vision. Distill, 2020. doi: 10.23915/distill.00029. https://distill.pub/2020/understanding-rl-vision. — [doi:10.23915/distill.00029](https://doi.org/10.23915/distill.00029) ↗
  28. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  29. M. L. Leavitt and A. Morcos. arXiv:2010.12016, 2020. — [arXiv:2010.12016](https://arxiv.org/abs/2010.12016) find
  30. Towards falsifiable interpretability research. arXiv preprint R. Massidda, A. Geiger, T. Icard, and D. Bacciu. Causal abstraction with soft interventions. arXiv preprint arXiv:2211.12270, 2022. — [arXiv:2211.12270](https://arxiv.org/abs/2211.12270) find
  31. T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, M. Wattenberg, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022. find
  32. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  33. A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv:1803.06959, 2018. — [arXiv:1803.06959](https://arxiv.org/abs/1803.06959) find
  34. N. Nanda, L. Chan, T. Liberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. nostalgebraist. interpreting gpt: the logit lens, 2020. URL https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logi… ↗
  35. C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev. The building blocks of interpretability. Distill, 3(3):e10, 2018. find
  36. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  37. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
  38. J. Pearl. Causality. Cambridge university press, 2009. find
  39. M. Phuong and M. Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238, 2022. — [arXiv:2207.09238](https://arxiv.org/abs/2207.09238) find
  40. A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
  41. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929–1958, 2014.
  42. N. Tinbergen. On aims and methods of ethology. Zeitschrift für tierpsychologie, 20(4):410–433, 1963. find
  43. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  44. A. Veit, M. J. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016. find
  45. J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, S. Sakenis, J. Huang, Y. Singer, and S. Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004.12265, 2020a. — [arXiv:2004.12265](https://arxiv.org/abs/2004.12265) find
  46. J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388–12401, 2020b. find
  47. K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. find
  48. B. Zhang and R. Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mcgrath-2023-hydra-effect-emergent-self-repair/