Elamur Research
☰
home / papers / 2022 / What learning algorithm is in-context learning? Investigations with linear models

What learning algorithm is in-context learning? Investigations with linear models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2211.15661

Summary

This paper asks how transformers manage in-context learning (ICL), meaning how they produce accurate predictions from labeled examples in the prompt without any weight updates. The authors test the idea that a trained transformer implicitly holds a small model in its activations and fits it to the examples as they arrive. Linear regression serves as the testbed. The study has three parts. First, explicit constructions show that decoder-only transformer layers can carry out gradient descent and closed-form ridge regression. Second, transformers trained directly on synthetic regression prompts are compared with classical estimators on underdetermined problems. Third, probes check whether the hidden states encode the intermediate quantities those estimators use. The trained models' predictions closely match gradient descent, ridge regression or least squares. Which estimator they match depends on model depth and label noise, and they approach the Bayes-optimal predictor once they are wide and deep enough. This matters because it gives an algorithmic, testable account of ICL: in the linear case, transformers appear to rediscover standard estimation algorithms.

Key points

  • Computational primitives: four operations (mov, mul, div, aff) can each be implemented by one transformer decoder layer.
  • Gradient descent and ridge constructions:
    • With hidden size O(d) and constant depth, a transformer can perform one gradient-descent step on d-dimensional regression.
    • With hidden size O(d²), it can update a ridge solution to include one new example.
    • Running n steps of either algorithm requires roughly n times as many layers.
  • Probing results: weight vectors and moment matrices can be decoded nonlinearly from late-layer activations, while control probes perform clearly worse.
  • Scaling sweep:
    • Setup: input dimensions d ∈ {1, 2, 4, 8, 12, 16, 20}, varying layers, hidden size and number of heads.
    • Heads: a single attention head suffices for the model to align with OLS rather than Ridge(λ=0.1).
    • Layers and hidden size: their requirements grow with d in a step-like way.
  • Not purely linear: by an R² analysis, ICL predictions are not fully linear in the underdetermined regime, so the learner's hypothesis class extends beyond linear functions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2211.15661 · arXiv 2211.15661

@misc{akyurek2022what,
  title = {{What learning algorithm is in-context learning? Investigations with linear models}},
  author = {Akyürek, Ekin and Schuurmans, Dale and Andreas, Jacob and Ma, Tengyu and Zhou, Denny},
  year = {2022},
  doi = {10.48550/arxiv.2211.15661},
  eprint = {2211.15661},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.15661}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. ArXiv preprint, abs/1610.01644, 2016. URL https://arxiv.org/abs/1610.01644. ↗ our card
  2. Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neura… find
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv preprint, abs/1607.06450, 2016. URL https://arxiv.org/abs/1607.06450. ↗ our card
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  5. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: circuits. Distill, 5(3):e24, 2020. find
  6. Stephanie CY Chan, Adam Santoro, Andrew K Lampinen, Jane X Wang, Aaditya Singh, Pierre H Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. ArXiv preprint, abs/2205.05055, 2022. URL https://arxiv.org/abs/2205.05055. ↗
  7. Chi Chen, Maosong Sun, and Yang Liu. Mask-align: Self-supervised neural word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4781–4791, Online… find
  8. Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 719–730, Dublin, Ireland, 2022. Association for Computational Lin… find
  9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗
  10. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedi… find
  11. PMLR, 2017. URL http://proceedings.mlr.press/v70/finn17a.html. ↗
  12. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. ArXiv, abs/2208.01066, 2022. find
  13. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). abs/1606.08415, 2016. URL https://arxiv.org/abs/1606.08415. ↗
  14. Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. find
  15. Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second, 2022. URL https://arxiv.org/abs/2207.01848. ↗
  16. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989. find
  17. Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34:14122–14134, 2021. find
  18. Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. ArXiv preprint, abs/2210.14215, 2022. URL https://arxiv.org/abs/2210.14215. — [arXiv:2210.14215](https://arxiv.org/abs/2210.14215) ↗
  19. David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. find
  20. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791–2809, Seattle, United States, 2022. Ass… find
  21. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2… find
  22. Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021. — [arXiv:2112.10510](https://arxiv.org/abs/2112.10510) find
  23. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, T. J. Henighan, Benjamin Mann, Amanda Askell, Yushi Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario Am… find
  24. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine… find
  25. PMLR, 2021. URL http://proceedings.mlr.press/v139/schlag21a.html. ↗
  26. ArXiv preprint, Juergen Schmidhuber, Jieyu Zhao, and Marco A Wiering. Simple principles of metalearning. 1996. find
  27. Jack Sherman and Winifred J Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1):124–127, 1950. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neur… find
  29. Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. ArXiv preprint, abs/2107.13163, 2021. URL https://arxiv.org/abs/2107.13163. — [arXiv:2107.13163](https://arxiv.org/abs/2107.13163) ↗
  30. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ArXiv preprint, abs/2111.02080, 2021. URL https: //arxiv.org/abs/2111.02080. find our card
  31. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ArXiv, abs/2111.02080, 2022. find our card
  32. Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  33. OpenReview.net, 2020. URL https://openreview.net/forum?id=ByxRM0Ntvr. ↗
  34. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… find
  35. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. find