Elamur Research⌕
☰
home / papers / 2022 / In-context Learning and Induction Heads

In-context Learning and Induction Heads

Paper · 2022 · Catherine Olsson, Nelson Elhage, Neel Nanda et al.

Summary

The authors from Anthropic search for the internal mechanism of transformer language models that is responsible for in-context learning. By this they mean a model's ability to better predict tokens that appear later in the context. The main object of study is induction heads. This is a circuit of attention heads that finds a previous occurrence of the current token A and proposes the token B that followed it last time. The team traced the training of 34 transformers of different sizes, from small attention-only models to large models with MLPs. It also performed more than 50,000 ablations of individual heads. The main observation is that a phase transition occurs early in training, visible as a "bump" in the loss curve. It is precisely at this point that induction heads appear and the capacity for in-context learning rises sharply at the same time. For small models the connection is confirmed causally, while for large models the authors have only correlational evidence. The work matters for mechanistic interpretability and AI safety. It links the level of individual circuits inside the network to the overall training dynamics and shows how model capabilities can emerge abruptly.

Key points

  • How the circuit works: first, a previous token head moves information about the preceding token into each position. Then the induction head itself uses this information to find the place where the current token has already occurred and copies what came after it. The authors hypothesize that in large models similar circuits also complete the pattern based on similarity, not only on exact matches (A* ≈ A, B* ≈ B).
  • Metric: the in-context learning score is the loss at the 500th token minus the loss at the 50th token, averaged over examples. The context length is 512 tokens. According to the authors, other indices do not change the conclusions.
  • Six lines of evidence: coincidence in time; a simultaneous shift of both phenomena when the architecture is changed; a drop in in-context learning after induction heads are knocked out at test time; examples of more abstract behavior by the same heads; mechanistic plausibility; smooth continuity from small models to large ones.
  • Scale of the experiments: 34 models, more than 50,000 ablations, each evaluated on 10,000 examples. For the small models 200 checkpoints each were saved, for the full-scale ones only 14–15.
  • At least two layers are required: the phase transition is observed only in models with two or more layers. An induction head works only in combination with another head from an earlier layer, so it cannot form in one-layer models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Few-shot learning · 45 works

Cite

Cite as Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish and Chris Olah. 2022. In-context Learning and Induction Heads. arXiv:2209.11895.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2209.11895 · arXiv 2209.11895

@misc{olsson2022context,
  title = {{In-context Learning and Induction Heads}},
  author = {Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish and Chris Olah},
  year = {2022},
  doi = {10.48550/arxiv.2209.11895},
  eprint = {2209.11895},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2209.11895}
}

References

47 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Language models are few-shot learners [PDF] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and others,, 2020. arXiv preprint arXiv:2005.14165.
  2. LaMDA: our breakthrough conversation technology [link] Collins, E. and Ghahramani, Z., 2021. find
  3. Evaluating large language models trained on code Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G. and others,, 2021. arXiv preprint arXiv:2107.03374.
  4. Towards a human-like open-domain chatbot Adiwardana, D., Luong, M., So, D.R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y. and others,, 2020. arXiv preprint arXiv:2001.09977.
  5. Scaling Language Models: Methods, Analysis & Insights from Training Gopher [PDF] Rae, J. W ., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G.v.d., Hendric… find
  6. Thread: Circuits [link] Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B. and Lim, S.K., 2020. Distill. find
  7. A Mathematical Framework for Transformer Circuits [HTML] Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei… find
  8. Grokking: Generalization beyond overfitting on small algorithmic datasets Power, A., Burda, Y., Edwards, H., Babuschkin, I. and Misra, V., 2022. arXiv preprint arXiv:2201.02177. find
  9. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [link] Pan, A., Bhatia, K. and Steinhardt, J., 2022. International Conference on Learning Representations. find
  10. Language Models are Unsupervised Multitask Learners [PDF] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I., 2019. find
  11. Calibrate Before Use: Improving Few-Shot Performance of Language Models Zhao, T.Z., Wallace, E., Feng, S., Klein, D. and Singh, S., 2021. find
  12. Making Pre-trained Language Models Better Few-shot Learners Gao, T., Fisch, A. and Chen, D., 2021. find
  13. Scaling laws for neural language models Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D., 2020. arXiv preprint arXiv:2001.08361.
  14. Risks from Learned Optimization in Advanced Machine Learning Systems Hubinger, E., Merwijk, C.v., Mikulik, V., Skalse, J. and Garrabrant, S., 2021. find
  15. Why does unsupervised pre-training help deep learning? [link] Erhan, D., Courville, A., Bengio, Y. and Vincent, P., 2010. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 201–208. find
  16. The Scaling Hypothesis [link] Branwen, G., 2020. find
  17. Shortformer: Better language modeling using shorter inputs Press, O., Smith, N.A. and Lewis, M., 2020. arXiv preprint arXiv:2012.15832. , , , , , p p — [arXiv:2012.15832](https://arxiv.org/abs/2012.15832) find
  18. A General Language Assistant as a Laboratory for Alignment Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N. and others,, 2021. arXiv preprint arXiv:2112.00861. find
  19. The Pile: An 800GB Dataset of Diverse Text for Language Modeling Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S. and Leahy, C., 2020. find
  20. Adversarial Reprogramming of Neural Networks Elsayed, G.F., Goodfellow, I. and Sohl-Dickstein, J., 2018. find
  21. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm Reynolds, L. and McDonell, K., 2021. find
  22. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Saxe, A.M., McClelland, J.L. and Ganguli, S., 2014. find
  23. Tensor2tensor transformer visualization [link] Jones, L., 2017. find
  24. A multiscale visualization of attention in the transformer model [PDF] Vig, J., 2019. arXiv preprint arXiv:1906.05714.
  25. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned [PDF] Voita, E., Talbot, D., Moiseev, F., Sennrich, R. and Titov, I., 2019. arXiv preprint arXiv:1905.09418.
  26. What does bert look at? an analysis of bert's attention [PDF] Clark, K., Khandelwal, U., Levy, O. and Manning, C.D., 2019. arXiv preprint arXiv:1906.04341. find
  27. Do attention heads in bert track syntactic dependencies? [PDF] Htut, P.M., Phang, J., Bordia, S. and Bowman, S.R., 2019. arXiv preprint arXiv:1911.12246. find
  28. Attention is not all you need: Pure attention loses rank doubly exponentially with depth [PDF] Dong, Y., Cordonnier, J. and Loukas, A., 2021. arXiv preprint arXiv:2103.03404. — [arXiv:2103.03404](https://arxiv.org/abs/2103.03404) find
  29. What Context Features Can Transformer Language Models Use? O'Connor, J. and Andreas, J., 2021. arXiv preprint arXiv:2106.08367.
  30. An Explanation of In-context Learning as Implicit Bayesian Inference Xie, S.M., Raghunathan, A., Liang, P. and Ma, T., 2021. arXiv preprint arXiv:2111.02080.
  31. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H. and Zettlemoyer, L., 2022. arXiv preprint arXiv:2202.12837.
  32. Reconciling modern machine-learning practice and the classical bias — variance trade-off Belkin, M., Hsu, D., Ma, S. and Mandal, S., 2019. Proceedings of the National Academy of Sciences, Vol 116(32), pp. 15849–15854. National Acad Sciences. find
  33. Deep double descent: Where bigger models and more data hurt Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B. and Sutskever, I., 2021. Journal of Statistical Mechanics: Theory and Experiment, Vol 2021(12), pp. 124003. IOP Publishing. find
  34. Future ML Systems Will Be Qualitatively Different [link] Steinhardt, J., 2021. Bounded Regret. find
  35. A mathematical theory of semantic development in deep neural networks Saxe, A.M., McClelland, J.L. and Ganguli, S., 2019. Proceedings of the National Academy of Sciences, Vol 116(23), pp. 11537–11546. National Acad Sciences. find
  36. Qualitatively characterizing neural network optimization problems Goodfellow, I.J., Vinyals, O. and Saxe, A.M., 2014. arXiv preprint arXiv:1412.6544. find
  37. Analyzing monotonic linear interpolation in neural network loss landscapes Lucas, J., Bae, J., Zhang, M.R., Fort, S., Zemel, R. and Grosse, R., 2021. arXiv preprint arXiv:2104.11044. find
  38. Geometry of neural network loss surfaces via random matrix theory Pennington, J. and Bahri, Y., 2017. International Conference on Machine Learning, pp. 2798–2806. find
  39. Visualizing the loss landscape of neural nets Li, H., Xu, Z., Taylor, G., Studer, C. and Goldstein, T., 2018. Advances in neural information processing systems, Vol 31. find
  40. Zoom In: An Introduction to Circuits Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. DOI: 10.23915/distill.00024.001 42. Convergent learning: Do different neural networks learn the same representations? Li, Y., Yosinski, J., Clune, J., Lipson, H., Hopcroft,… — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) find
  41. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability [PDF] Raghu, M., Gilmer, J., Yosinski, J. and Sohl-Dickstein, J., 2017. Advances in Neural Information Processing Systems 30, pp. 6078–6087. Curran Associates, Inc. find
  42. Similarity of neural network representations revisited [PDF] Kornblith, S., Norouzi, M., Lee, H. and Hinton, G., 2019. arXiv preprint arXiv:1905.00414.
  43. High-Low Frequency Detectors Schubert, L., Voss, C., Cammarata, N., Goh, G. and Olah, C., 2021. Distill. DOI: 10.23915/distill.00024.005 46. Performance-optimized hierarchical models predict neural responses in higher visual cortex Yamins, D.L., Hong, H., Cadieu, C.F., Solomon, E.A., Seibert, D. and… — [doi:10.23915/distill.00024.005](https://doi.org/10.23915/distill.00024.005) find
  44. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream Güçlü, U. and van Gerven, M.A., 2015. Journal of Neuroscience, Vol 35(27), pp. 10005–10014. Soc Neuroscience. find
  45. Seeing it all: Convolutional network layers map the function of the human visual system Eickenberg, M., Gramfort, A., Varoquaux, G. and Thirion, B., 2017. NeuroImage, Vol 152, pp. 184–194. Elsevier. find
  46. Multimodal Neurons in Artificial Neural Networks Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A. and Olah, C., 2021. Distill. DOI: 10.23915/distill.00030 50. Neural machine translation by jointly learning to align and translate Bahdanau, D., Cho, K. and Bengio, Y.… find
  47. Listen, attend and spell Chan, W., Jaitly, N., Le, Q. V . and Vinyals, O., 2015. arXiv preprint arXiv:1508.01211. \#\# Acknowledgments In writing this paper, our thinking and exposition was greatly clarified by detailed correspondence with Sam Bowman, Paul Christiano, Aidan Gomez, Dan Hendrycks, Jacob… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/