Elamur Research⌕
☰
home / papers / 2024 / LLM-SR: Scientific Equation Discovery via Programming with Large Language Models

LLM-SR: Scientific Equation Discovery via Programming with Large Language Models

Paper · 2024 · Parshin Shojaee, Kazem Meidani, Shashank Gupta et al.

Summary

Symbolic regression aims to recover compact, interpretable formulas from observations. The search space is combinatorially huge (the task is NP-hard), and classic tools such as genetic programming over expression trees ignore the domain knowledge that human scientists bring. LLM-SR addresses this by asking a large language model to write candidate equations as Python functions. Each candidate is a skeleton whose numeric constants are left as placeholders, and the prompt describes the problem and its variables in plain language. An optimizer then fits the constants to data and scores the result. The best-scoring programs go into an experience buffer, which supplies in-context examples for the next round, forming an evolutionary refinement loop. Popular benchmarks like Feynman may already be memorized by LLMs, so the authors built four new problems with synthetic twists. Across these problems, LLM-SR beat leading symbolic regression baselines, with the largest gains on out-of-domain data, and it reached good equations in fewer iterations. This suggests LLM priors can make automated equation discovery both more efficient and more physically meaningful.

Key points

  • Method: The framework has three stages: hypothesis generation by the LLM, data-driven evaluation, and experience management through a diverse buffer of strong candidates. Constants are fitted with Torch+Adam or NumPy+BFGS.
  • Backbone LLMs: GPT-3.5-turbo and Mixtral-8x7B. The baselines include DSR, uDSR and PySR.
  • Benchmarks: Four custom problems were designed to avoid recitation:
    • two nonlinear oscillators (Oscillation 1 and 2),
    • E. coli growth rate (biology),
    • a temperature-dependent stress-strain relation (materials science).
  • Flexible representation: Because candidates are programs, they can use if-else logic or sigmoid relaxations to capture piecewise behavior, such as the plastic region in stress-strain data. Expression-tree methods struggle with this.
  • Ablations: Removing any of these components hurts performance: problem descriptions, program representation, data feedback, or iterative refinement. Code and data are available at github.com/deep-symbolic-mathematics/LLM-SR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputational Theory and Mathematics · 496 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani and Chandan K Reddy. 2024. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models. arXiv:2404.18400.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.18400 · arXiv 2404.18400

@misc{shojaee2024llm,
  title = {{LLM-SR: Scientific Equation Discovery via Programming with Large Language Models}},
  author = {Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani and Chandan K Reddy},
  year = {2024},
  doi = {10.48550/arxiv.2404.18400},
  eprint = {2404.18400},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.18400}
}

References

50 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B.S. Aakash, JohnPatrick Connors, and Michael D. Shields. Stress-strain data for aluminum 6061-t651 from 9 lots at 6 temperatures under uniaxial and plane strain tension. Data in Brief, 25: 104085, 2019. ISSN 2352-3409. doi: https://doi.org/10.1016/j.dib.2019.104085. — [doi:10.1016/j.dib.2019.104085](https://doi.org/10.1016/j.dib.2019.104085) ↗
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  3. Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023. — [arXiv:2311.07361](https://arxiv.org/abs/2311.07361) find
  4. Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Researc… find
  5. Jure Brence, Ljupčo Todorovski, and Sašo Džeroski. Probabilistic grammars for equation discovery. Knowledge-Based Systems, 224:107077, 2021. ISSN 0950-7051. doi: https://doi.org/10.1016/j. knosys.2021.107077. — [doi:10.1016/j](https://doi.org/10.1016/j) find
  6. Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  7. Angelica Chen, David Dohan, and David So. Evoprompting: Language models for code-level neural architecture search. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7787–7817. Curran Associates, Inc., 2023… find
  8. Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582, 2023.
  9. Roger Fletcher. Practical Methods of Optimization. John Wiley & Sons, New York, NY, USA, second edition, 1987. find
  10. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024. find
  11. Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better. arXiv preprint arXiv:2207.14502, 2022.
  12. Heng Ji, Qingyun Wang, Doug Downey, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. In ACL Anthology: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299. University of Illinois Urbana-Champaign/C…
  13. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  14. Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and Francois Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, 2022.
  15. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  16. John R. Koza. Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4(2):87–112, Jun 1994. ISSN 1573-1375. doi: 10.1007/BF00175355. — [doi:10.1007/bf00175355](https://doi.org/10.1007/bf00175355) find
  17. William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing System…
  18. Mikel Landajuela, Chak Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Terrell N. Mundhenk, Garrett Mulcahy, and Brenden K. Petersen. A unified framework for deep symbolic regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural… find
  19. Robert Tjarko Lange, Yingtao Tian, and Yujin Tang. Large language models as evolution strategies. arXiv preprint arXiv:2402.18381, 2024.
  20. Pat Langley. Data-driven discovery of physical laws. Cognitive Science, 5(1):31–54, 1981. find
  21. Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331–366. Springer, 2023. find
  22. Michael Y Li, Emily B Fox, and Noah D Goodman. Automated statistical model discovery with language models. arXiv preprint arXiv:2402.17879, 2024.
  23. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. find
  24. Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning Representations, 2023. find
  25. Pingchuan Ma, Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Joshua B Tenenbaum, Daniela Rus, Chuang Gan, and Wojciech Matusik. Llm and simulation as bilevel optimizers: A new paradigm to advance physical scientific discovery. arXiv preprint arXiv:2405.09783, 2024.
  26. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. find
  27. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Data-driven discovery with large generative models. arXiv preprint arXiv:2402.13610, 2024.
  28. Michael de la Maza and Bruce Tidor. An analysis of selection procedures with particular attention paid to proportional and boltzmann selection. In Proceedings of the 5th International Conference on Genetic Algorithms, pp. 124–131, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. ISBN 15… find
  29. Kazem Meidani, Parshin Shojaee, Chandan K Reddy, and Amir Barati Farimani. Snip: Bridging mathematical symbolic and numeric realms with unified pre-training. In The Twelfth International Conference on Learning Representations, 2023.
  30. Elliot Meyerson, Mark J Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170, 2023. — [arXiv:2302.12170](https://arxiv.org/abs/2302.12170) find
  31. Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Infor… find
  32. Brenden K Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021. find
  33. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. arXiv preprint arXiv:2311.05965, 2023.
  34. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Natu…
  35. Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science Advance, 324(5923):81–85, 2009. ISSN 0036-8075. doi: 10.1126/science.1165893. — [doi:10.1126/science.1165893](https://doi.org/10.1126/science.1165893) find
  36. Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816, 2023.
  37. Parshin Shojaee, Kazem Meidani, Amir Barati Farimani, and Chandan Reddy. Transformer-based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2024. find
  38. Fangzheng Sun, Yang Liu, Jian-Xun Wang, and Hao Sun. Symbolic physics learner: Discovering governing equations via monte carlo tree search. In The Eleventh International Conference on Learning Representations, 2023. find
  39. Ljupco Todorovski and Saso Dzeroski. Declarative bias in equation discovery. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pp. 376–384, San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc. ISBN 1558604863. find
  40. Ljupčo Todorovski and Sašo Džeroski. Integrating Domain Knowledge in Equation Discovery, pp. 69–97. Springer Berlin Heidelberg, Berlin, Heidelberg, 2007. ISBN 978-3-540-73920-3. doi: 10.1007/978-3-540-73920-3\\ 4. — [doi:10.1007/978-3-540-73920-3\](https://doi.org/10.1007/978-3-540-73920-3\) find
  41. Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. doi: 10.1126/sciadv.aay2631.
  42. Marco Virgolin and Solon P Pissis. Symbolic regression is NP-hard. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
  43. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsund… find
  44. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660, 2023b. find
  45. Xingyu Wu, Sheng-hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Evolutionary computation in the era of large language model: Survey and roadmap. arXiv preprint arXiv:2401.10034, 2024. find
  46. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, 2023. find
  47. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023a.
  48. Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. In Neural Information Processing Systems (NeurIPS), 2023b.
  49. Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie. Can gpt-4 perform neural architecture search? arXiv preprint arXiv:2304.10970, 2023a.
  50. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.07984, 2023b. — [arXiv:2310.07984](https://arxiv.org/abs/2310.07984) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shojaee-2024-llm-sr-scientific-equation/