Elamur Research⌕
☰
home / papers / 2024 / Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective

Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective

Paper · 2024 · Fabian Falck, Ziyu Wang and Chris Holmes

Summary

The paper asks whether in-context learning (ICL) in large language models behaves like Bayesian inference when the prompt contains exchangeable data, a view many earlier works have assumed. The authors approach this through the martingale property. A Bayesian learner must satisfy it: if the model imputes missing samples from itself and then predicts, the averaged prediction should equal the prediction made from the observed data alone. They argue that this property is needed for order-independent, unambiguous predictions and for splitting uncertainty into epistemic and aleatoric parts. From the theory they build practical diagnostics and test statistics. They also compare how fast LLM uncertainty shrinks with more data against a Bayesian reference. The tests run on deliberately simple synthetic tasks with known data-generating processes. Across Llama-2, Mistral, GPT-3, GPT-3.5 and GPT-4, they find violations of the martingale property and non-Bayesian uncertainty scaling. The authors therefore reject the claim that ICL is Bayesian and warn against relying on LLM ICL in safety-critical, exchangeable-data settings such as clinical trial imputation.

Key points

  • The motivating example is an interim analysis of a two-arm RCT (100 patients, 50 per arm), in which an LLM imputes the remaining patients. If the martingale property fails, the result can depend on which patient is imputed first.
  • When a model shifts its own predictions on average by querying itself, the authors call this "introspective hallucination." The violations show up as drift over longer sampled paths.
  • The martingale property approximately holds for all LLMs only in a Bernoulli setting with θ = 0.5 and a short horizon (m = n/2). Epistemic uncertainty scaling (statistic T3) is examined in that setting and compared with a reference Bayesian model and α-fractional Bayesian posteriors (α = 0.5 and α = 2).
  • Llama-2-7B and GPT-3.5 shrink uncertainty more slowly than a Bayesian model, and GPT-3.5 is over-confident when samples are few. GPT-3-170B and Mistral-7B come closer to Bayesian scaling and also adhere best to the martingale property.
  • The paper was published at ICML 2024 (University of Oxford). The authors name the restriction to small synthetic datasets as the main limitation and point to tabular-data prediction and synthetic data generation as uses for the diagnostics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom138 works · 161,448 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Fabian Falck, Ziyu Wang and Chris Holmes. 2024. Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective. arXiv:2406.00793.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.00793 · arXiv 2406.00793

@misc{falck2024is,
  title = {{Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective}},
  author = {Fabian Falck, Ziyu Wang and Chris Holmes},
  year = {2024},
  doi = {10.48550/arxiv.2406.00793},
  eprint = {2406.00793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.00793}
}

References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022.
  2. Hahn, M. and Goyal, N. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971, 2023. — [arXiv:2303.07971](https://arxiv.org/abs/2303.07971) find
  3. Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637, 2023. — [arXiv:2306.04637](https://arxiv.org/abs/2306.04637) find
  4. Hämäläinen, P., Tavast, M., and Kunnari, A. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–19, 2023. find
  5. Berti, P., Pratelli, L., and Rigo, P. Limit theorems for a class of identically distributed random variables. The Annals of Probability, 32(3), July 2004. ISSN 0091-1798. doi: 10.1214/009117904000000676. — [doi:10.1214/009117904000000676](https://doi.org/10.1214/009117904000000676) find
  6. Bhattacharya, A., Pati, D., and Yang, Y. Bayesian fractional posteriors. Annals of Statistics, 47(1):39–66, 2019. find
  7. Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with NumPy. Nature, 585(7825):357–362, 2020.
  8. Biewald, L. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com. ↗
  9. Hunter, J. D. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3):90–95, 2007. doi: 10.1109/MCSE.2007.55. — [doi:10.1109/mcse.2007.55](https://doi.org/10.1109/mcse.2007.55) find
  10. Borisov, V., Seßler, K., Leemann, T., Pawelczyk, M., and Kasneci, G. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280, 2022. — [arXiv:2210.06280](https://arxiv.org/abs/2210.06280) find
  11. Huszár, F. Implicit bayesian inference in large language models. https://www.inference.vc/implicit-bayesian-inference-in-sequence-models/, 2022. ↗
  12. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  13. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  14. Jiang, H. A latent space theory for emergent abilities in large language models. arXiv preprint arXiv:2304.09960, 2023. — [arXiv:2304.09960](https://arxiv.org/abs/2304.09960) find
  15. De Finetti, B. Funzione caratteristica di un fenomeno aleato-rio. In Atti del Congresso Internazionale dei Matematici: Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., et al. Time-llm: Time series forecasting by reprogramming large language model… — Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
  16. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Researc… find
  17. Kalai, A. T. and Vempala, S. S. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14648, 2023. — [arXiv:2311.14648](https://arxiv.org/abs/2311.14648) find
  18. Raventós, A., Paul, M., Chen, F., and Ganguli, S. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. arXiv preprint arXiv:2306.15063, 2023. — [arXiv:2306.15063](https://arxiv.org/abs/2306.15063) find
  19. Kallenberg, O. Foundations of modern probability, volume 2. Springer, 1997. find
  20. Li, Z., Zhu, H., Lu, Z., and Yin, M. Synthetic data generation with large language models for text classification: Potential and limitations. arXiv preprint arXiv:2310.07849, 2023.
  21. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. Model dementia: Generated data makes models forget. arXiv e-prints, pp. arXiv–2305, 2023. find
  22. Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
  23. Singh, A. K., Chan, S. C., Moskovitz, T., Grant, E., Saxe, A. M., and Hill, F. The transient nature of emergent in-context learning in transformers. arXiv preprint arXiv:2311.08360, 2023. — [arXiv:2311.08360](https://arxiv.org/abs/2311.08360) find
  24. Manikandan, H., Jiang, Y., and Kolter, J. Z. Language models are weak learners, June 2023. URL http://arxiv.org/abs/2306.14101. arXiv:2306.14101 [cs]. — [arXiv:2306.14101](https://arxiv.org/abs/2306.14101) ↗
  25. Tang, R., Han, X., Jiang, X., and Hu, X. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360, 2023. — [arXiv:2303.04360](https://arxiv.org/abs/2303.04360) find
  26. Mei, Y., Song, S., Fang, C., Yang, H., Fang, J., and Long, J. Capturing semantics for imputation with pre-trained language models. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 61–72. find
  27. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  28. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021. tqdm contributors. Imageio. https://github.com/tqdm/tqdm, 2022. ↗
  29. OpenAI. Gpt-4 technical report, 2023.
  30. Van der Vaart, A. W. Asymptotic statistics, volume 3. Cambridge university press, 2000. find
  31. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
  32. Van Rossum, G. The Python Library Reference, release 3.8.2. Python Software Foundation, 2020. find
  33. Veselovsky, V., Ribeiro, M. H., Arora, A., Josifoski, M., Anderson, A., and West, R. Generating faithful synthetic data with large language models: A case study in computational social science. arXiv preprint arXiv:2305.15041, 2023. — [arXiv:2305.15041](https://arxiv.org/abs/2305.15041) find
  34. Panwar, M., Ahuja, K., and Goyal, N. In-context learning through the bayesian prism. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=HX5ujdsSon. ↗
  35. Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  36. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-perfor… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  37. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  38. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 – 61, 2010. doi: 10.25080/Majora-92bf1922-00a.
  39. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-ofthe-art natural l… find
  40. Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., and Morency, L.-P. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04714, 2022. find
  41. Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021.
  42. Yan, J., Zheng, B., Xu, H., Zhu, Y., Chen, D., Sun, J., Wu, J., and Chen, J. Making pre-trained language models great on tabular prediction. In International Conference on Learning Representations, 2024. find
  43. Ye, N., Yang, H., Siah, A., and Namkoong, H. Pre-training and in-context learning IS bayesian inference a la de finetti. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https://openreview.net/forum? id=ttupfosvgx. ↗
  44. Zellner, A. Optimal information processing and bayes’s theorem. The American Statistician, 42(4):278–280, 1988. find
  45. Zhang, L., McCoy, R. T., Sumers, T. R., Zhu, J.-Q., and Griffiths, T. L. Deep de finetti: Recovering topic distributions from large language models. arXiv preprint arXiv:2312.14226, 2023a. — [arXiv:2312.14226](https://arxiv.org/abs/2312.14226) find
  46. Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420, 2023b. — [arXiv:2305.19420](https://arxiv.org/abs/2305.19420) find
  47. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  48. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697–12706. PMLR, 2021.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/falck-2024-context-learning-large-language-models/