Elamur Research
☰
home / papers / 2021 / An Explanation of In-context Learning as Implicit Bayesian Inference

An Explanation of In-context Learning as Implicit Bayesian Inference

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2111.02080

Summary

The paper asks why large LMs such as GPT-3 can pick up a task from input-output examples in a prompt, even though they were never trained to learn from examples. The authors suggest that pretraining text has long-range coherence: to predict the next token well, an LM must implicitly infer a latent document-level "concept." At test time, in-context learning happens when the model infers the concept that the prompt examples share. In this view, prompting amounts to implicit Bayesian inference over concepts.

The authors formalize this with pretraining data drawn from a mixture of HMMs. They prove that in-context prediction becomes asymptotically optimal when each example carries more signal about the concept than the error introduced by the prompt/pretraining mismatch. To test the theory, they build GINC, a small synthetic dataset. Transformers and LSTMs trained on it learn in context and reproduce effects seen in large models. The work offers a tractable, small-scale testbed and a theoretical account that could guide better pretraining and prompting.

Key points

  • Theory: in-context error is shown to fall as more examples are added and as each example gets longer. This means the input tokens themselves, not only the input-to-label mapping, help identify the concept. The standard Bernstein-von Mises result does not apply here, because prompt examples are neither independent nor drawn from the pretraining distribution.
  • GINC experiments: they compare Transformers (4, 12 and 16 layers; 29M, 85M and 115M parameters) and a 28M LSTM. With vocabulary size 50, in-context accuracy climbs from about 60% (4-layer) to about 85% (16-layer), even though the 12- and 16-layer models reach nearly identical pretraining loss (~1.31). The LSTM reaches about 96%.
  • Ablations: removing the latent concept structure from the pretraining data prevents in-context learning from emerging.
  • Mirrored real-world quirks:
    • Accuracy varies widely depending on the order of the examples.
    • Zero-shot can beat few-shot in some settings, such as low-entropy transitions (temperature 0.01, 12 concepts, vocab 100). This echoes GPT-3's behavior on LAMBADA.
  • Availability: code and data are released on GitHub and CodaLab.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

248 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 225 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.02080 · arXiv 2111.02080

@misc{xie2021explanation,
  title = {{An Explanation of In-context Learning as Implicit Bayesian Inference}},
  author = {Xie, Sang Michael and Raghunathan, Aditi and Liang, Percy and Ma, Tengyu},
  year = {2021},
  doi = {10.48550/arxiv.2111.02080},
  eprint = {2111.02080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.02080}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 37(6):1554–1563, 1966. find
  2. D. Blei, Andrew Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research (JMLR), 3:993–1022, 2003. find our card
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR), 2020. find
  5. A. P. Dempster, Laird N. M., and Rubin D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1):1–38, 1977. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171–4186, 2019. find
  7. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv, 2021. find our card
  8. Zoubin Ghahramani and Michael Jordan. Factorial hidden Markov models. Machine Learning, 29: 245–273, 1997. find
  9. Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hidden topic Markov models. In Artificial Intelligence and Statistics (AISTATS), 2007. find
  10. M. Gunst and O. Shcherbakova. Asymptotic behavior of Bayes estimators for hidden Markov models with application to ion channels. Mathematical Methods of Statistics, 17, 2008. find
  11. Keith W. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97–109, 1970. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. find
  13. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020. find
  14. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right, 2021. find our card
  15. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? In Association for Computational Linguistics (ACL), 2020. find our card
  16. Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183–233, 1999. find
  17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL), 2017. find our card
  18. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. find
  19. B.J.K. Kleijn and A.W. van der Vaart. The Bernstein-von mises theorem under misspecification. Electronic Journal of Statistics, 6, 2012. find
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. find
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACL), 2020. find our card
  22. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021. find
  23. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs, August 2021. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  25. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. find
  26. Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087–1092, 1953. find
  27. Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Association for Computational Linguistics (ACL), 2016. find our card
  28. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR MATH AI Workshop, 2021. find
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  30. Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017. find
  31. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… find our card
  32. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. In European Association for Computational Linguistics (EACL), 2021. find our card
  33. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Eliciting knowledge from language models using automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020. find
  34. Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation, 26, 2007. find
  35. A. W. van der Vaart. Asymptotic statistics. Cambridge University Press, 1998. find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. find
  37. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. ↗
  38. Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. arXiv, 2021a. find
  39. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. arXiv, 2021b. find
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. find our card
  41. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017. find
  42. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML), 2021. find our card
  43. Bernardo Ávila Pires and Csaba Szepesvári. Multiclass classification calibration functions. arXiv, 2016. find