Elamur Research⌕
☰
home / papers / 2024 / Automated Statistical Model Discovery with Language Models

Automated Statistical Model Discovery with Language Models

Paper · 2024 · Michael Y. Li, Emily B. Fox and Noah D. Goodman

Summary

Building a good statistical model usually requires both a skilled modeler and a domain expert. Earlier automation systems, such as the Automatic Statistician, only worked inside a hand-designed domain-specific language (DSL) with a fixed search routine. The authors propose BoxLM, which lets large language models fill both roles within Box's Loop. A proposal LM writes candidate models as probabilistic programs. It conditions on the data, its metadata, the best earlier programs and textual feedback. Each program is fitted with generic inference and scored, for instance by ELPD LOO. A critic LM then reads posterior predictive statistics and turns them into plain-language advice for the next iteration. The evaluation covers three scenarios: search within a DSL of Gaussian process kernels, open-ended modeling of real datasets, and refining classic expert models under constraints stated in natural language. BoxLM performs on par with the Automatic Statistician and with programs written by human experts, and it extends Lotka-Volterra dynamics in interpretable ways. Because no DSL or custom search has to be engineered, model discovery could become more accessible.

Key points

  • Published at ICML 2024 (PMLR 235) by Stanford researchers. The system, BoxLM, generates m programs per round, keeps the top-k as exemplars, and uses critic-LM feedback to steer the next round.
  • In an illustrative dataset of 27 dugongs (age vs. length), the critic observes that programs using the von Bertalanffy growth function obtain better LOO scores. It also recommends informative, biology-based priors.
  • In the Stan experiments, scores averaged over 3 rounds of Box's Loop improve from round to round on average, though not monotonically for every individual run.
  • On Lotka-Volterra predator-prey data, all BoxLM variants (No-WS, WS-Constraint, WS-No Constraint) beat a standard Lotka-Volterra baseline and a hybrid neural ODE baseline on test MAE, averaged over three runs. Under an interpretability constraint, BoxLM extended the Rosenzweig–MacArthur model with Holling type II response by making handling time an MLP function of prey density.
  • Stated limitations:
    • only static, one-dimensional datasets were studied;
    • criticism statistics (residuals, posterior predictive mean) were fixed in advance;
    • the method relies on in-context learning rather than fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Michael Y. Li, Emily B. Fox and Noah D. Goodman. 2024. Automated Statistical Model Discovery with Language Models. arXiv:2402.17879.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.17879 · arXiv 2402.17879

@misc{li2024automated,
  title = {{Automated Statistical Model Discovery with Language Models}},
  author = {Michael Y. Li, Emily B. Fox and Noah D. Goodman},
  year = {2024},
  doi = {10.48550/arxiv.2402.17879},
  eprint = {2402.17879},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.17879}
}

References

44 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abril-Pla, O., Andreani, V., Carroll, C., Dong, L., Fonnesbeck, C. J., Kochurov, M., Kumar, R., Lao, J., Luhmann, C. C., Martin, O. A., Osthege, M., Vieira, R., Wiecki, T., and Zinkov, R. PyMC: a modern, and comprehensive probabilistic programming framework in python. PeerJ Computer Science, 9, 2023… find
  2. Achiam, O. J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonof… find
  3. Baum, L. E. and Petrie, T. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 37(6):1554–1563, 1966. find
  4. Blei, D. M. Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, 1(1):203–232, 2014. find
  5. Bommarito, M. J. and Katz, D. M. GPT Takes the Bar Exam. ArXiv, abs/2212.14402, 2022. find
  6. Bongard, J. C. and Lipson, H. Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104:9943 – 9948, 2007. find
  7. Box, G. E. P. and Hunter, W. G. A useful method for model-building. Technometrics, 4:301–318, 1962. find
  8. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
  9. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — Language Models are Few-Shot Learners
  10. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., S… — Evaluating Large Language Models Trained on Code
  11. Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 6572–6583, Red Hook, NY, USA, 2018. Curran Associates Inc. find
  12. Duvenaud, D., Lloyd, J., Grosse, R., Tenenbaum, J., and Zoubin, G. Structure discovery in nonparametric regression through compositional kernel search. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine… find
  13. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. find
  14. Gelman, A. and Rubin, D. B. Inference from iterative simulation using multiple sequences. Statistical Science, 7(4): 457–472, 1992. ISSN 08834237. find
  15. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. Bayesian data analysis, third edition. 2013. find
  16. Goodman, N. D., Mansinghka, V. K., Roy, D. M., Bonawitz, K., and Tenenbaum, J. B. Church: a language for generative models. In Conference on Uncertainty in Artificial Intelligence, 2008. find
  17. Grosse, R. B. Model selection in compositional spaces. 2014. find
  18. Homan, M. D. and Gelman, A. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593–1623, jan 2014. ISSN 1532-4435. find
  19. Kalman, R. E. A new approach to linear filtering and prediction problems. ASME Journal of Basic Engineering, 82(1):35–45, March 1960. find
  20. Kidger, P. On Neural Differential Equations. PhD thesis, University of Oxford, 2021. find
  21. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
  22. Lee, P., Bubeck, S., and Petro, J. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine, 388(13):1233–1239, 2023. doi: 10.1056/NEJMsr2214184. PMID: 36988602. — [doi:10.1056/nejmsr2214184](https://doi.org/10.1056/nejmsr2214184) find
  23. Linka, K., St. Pierre, S. R., and Kuhl, E. Automated model discovery for human brain using constitutive artificial neural networks. Acta Biomaterialia, 160:134–151, 2023. ISSN 1742-7061. doi: https://doi.org/10.1016/j.actbio.2023.01.055. — [doi:10.1016/j.actbio.2023.01.055](https://doi.org/10.1016/j.actbio.2023.01.055) ↗
  24. Lloyd, J. R., Duvenaud, D. K., Grosse, R. B., Tenenbaum, J. B., and Ghahramani, Z. Automatic construction and natural-language description of nonparametric regression models. In AAAI Conference on Artificial Intelligence, 2014. find
  25. Magnusson, M., Bürkner, P., and Vehtari, A. posteriordb: a set of posteriors for Bayesian inference and probabilistic programming, October 2023. find
  26. McKinney, B. A., Crowe, J. E., Voss, H. U., Crooke, P. S., Barney, N., and Moore, J. H. Hybrid grammar-based approach to nonlinear dynamical system identification from biological time series. Phys. Rev. E, 73:021912, Feb 2006. doi: 10.1103/PhysRevE.73.021912. — [doi:10.1103/physreve.73.021912](https://doi.org/10.1103/physreve.73.021912) find
  27. Miller, A. C., Foti, N. J., and Fox, E. Learning insulin-glucose dynamics in the wild. In Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pp. 172–197. PMLR, 07–08 Aug 2020. find
  28. Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. ArXiv, abs/1905.10437, 2019. — [arXiv:1905.10437](https://arxiv.org/abs/1905.10437) find
  29. Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., and Ren, X. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference on Learning Representation… find
  30. Rabiner, L. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286, 1989. doi: 10.1109/5.18626. — [doi:10.1109/5.18626](https://doi.org/10.1109/5.18626) find
  31. Rosenzweig, M. and MacArthur, R. Graphical representation and stability conditions of predator-prey interaction. American Naturalist, 97:209–223, 1963. find
  32. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M. P., Ferrer, C. C., Grattafiori, A., Xiong, W., D’efossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., an… — Code Llama: Open Foundation Models for Code
  33. Saad, F. A., Patton, B., Hoffman, M., Saurous, R. A., and Mansinghka, V. K. Sequential monte carlo learning for time series structure discovery. In International Conference on Machine Learning, 2023. find
  34. Schick, T., Dwivedi-Yu, J., Dessı̀, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761, 2023.
  35. Schmidt, M. and Lipson, H. Distilling free-form natural laws from experimental data. Science, 324(5923):81–85, 2009. doi: 10.1126/science.1165893. — [doi:10.1126/science.1165893](https://doi.org/10.1126/science.1165893) find
  36. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. van de Meent, J.-W., Paige, B., Yang, H., and Wood, F. An introduction to probabilistic pro…
  37. Vehtari, A., Gelman, A., and Gabry, J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5): 1413–1432, sep 2017. ISSN 0960-3174. doi: 10.1007/s11222-016-9696-4. von Bertalanffy, L. Problems of organic growth. Nature, 163:156–158, 1949. — [doi:10.1007/s11222-016-9696-4](https://doi.org/10.1007/s11222-016-9696-4) find
  38. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024. find
  39. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  40. Wilson, A. G. and Adams, R. P. Gaussian process kernels for pattern discovery and extrapolation. In Proc. ICML, 2013. find
  41. Wong, L. S., Grand, G., Lew, A. K., Goodman, N. D., Mansinghka, V. K., Andreas, J., and Tenenbaum, J. B. From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought. ArXiv, abs/2306.12672, 2023. find
  42. Wood, F., van de Meent, J. W., and Mansinghka, V. A new approach to probabilistic programming inference. In Proceedings of the 17th International conference on Artificial Intelligence and Statistics, pp. 1024–1032, 2014. find
  43. Wu, Y., Jia, F., Zhang, S., Li, H., Zhu, E., Wang, Y., Lee, Y. T., Peng, R., Wu, Q., and Wang, C. An Empirical Study on Challenging Math Problem Solving with GPT-4, 2023. find
  44. Zhong, R., Zhang, P., Li, S., Ahn, J., Klein, D., and Steinhardt, J. Goal driven discovery of distributional differences via language descriptions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2024-automated-statistical-model-discovery/