Elamur Research⌕
☰
home / papers / 2025 / LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

Paper · 2025 · Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani et al.

Summary

Benchmarks for symbolic regression, such as Feynman, SRBench and SRSD, mostly contain textbook physics laws. LLM-based equation discovery methods can therefore score well by recalling a familiar formula from variable names rather than by inferring it from data. The authors demonstrate this: simple Llama-3.1-8B sampling on Feynman tasks reaches low error almost immediately.

To address this, they built LLM-SRBench, a set of 239 problems in two groups:

  • LSR-Transform takes Feynman equations, picks a different input variable as the new target, and uses SymPy to re-solve each equation into an unfamiliar form.
  • LSR-Synth combines known scientific terms with novel but plausible synthetic terms. Numerical solvers check that each problem is solvable.

Methods are scored on data fit (in-domain and out-of-domain), symbolic correctness and compute cost. Across several discovery methods with open and closed LLM backbones, the strongest system recovers only about 31.5% of equations. This shows that current LLM-driven scientific discovery remains weak once memorization is ruled out.

Key points

  • 239 tasks: 111 in LSR-Transform (derived from the 100-equation Feynman benchmark) and 128 in LSR-Synth.
  • LSR-Synth covers four domains: chemistry (36), biology (24), physics (43) and material science (25).
  • Top results: 31.5% symbolic accuracy on LSR-Transform and 28.1% on LSR-Synth.
  • The evaluation protocol accepts hypotheses as expression strings or programs. It measures in-domain accuracy, out-of-domain generalization, symbolic accuracy (judged by an LLM evaluator) and efficiency.
  • Published at ICML 2025 (arXiv:2504.10415); code and data are at github.com/deep-symbolic-mathematics/llm-srbench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksComputational Theory and Mathematics · 496 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan and Chandan K Reddy. 2025. LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. arXiv:2504.10415.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.10415 · arXiv 2504.10415

@misc{shojaee2025llm,
  title = {{LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models}},
  author = {Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan and Chandan K Reddy},
  year = {2025},
  doi = {10.48550/arxiv.2504.10415},
  eprint = {2504.10415},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.10415}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Biggio, L., Bendinelli, T., Neitz, A., Lucchi, A., and Parascandolo, G. Neural symbolic regression that scales. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 936–945. PMLR, 18–24 Ju… find
  2. Langley, P. Data-driven discovery of physical laws. Cognitive Science, 5(1):31–54, 1981. find
  3. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  4. Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. Scienceagentbench: Toward rigorous assessment of language Ma, P., Wang, T.-H., Guo, M., Sun, Z., Tenenbaum, J. B., Rus, D., Gan, C., and Matusik, W. LLM and simulation as bilevel optimizers: A new pa… — LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery
  5. Schmidt, M. and Lipson, H. Distilling free-form natural laws from experimental data. Science Advance, 324 (5923):81–85, 2009. ISSN 0036-8075. doi: 10.1126/ science.1165893. find
  6. Shojaee, P., Meidani, K., Barati Farimani, A., and Reddy, C. Transformer-based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2024a. find
  7. Majumder, B. P., Surana, H., Agarwal, D., Hazra, S., Sabharwal, A., and Clark, P. Data-driven discovery with large generative models. arXiv preprint arXiv:2402.13610, 2024a.
  8. Shojaee, P., Meidani, K., Gupta, S., Farimani, A. B., and Reddy, C. K. Llm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404.18400, 2024b.
  9. Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725, 2024b.
  10. Strogatz, S. H. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering. CRC press, 2018. find
  11. Matsubara, Y., Chiba, N., Igarashi, R., Tatsunori, T., and Ushiku, Y. Rethinking symbolic regression datasets and benchmarks for scientific discovery. arXiv preprint arXiv:2206.10540, 2022. — [arXiv:2206.10540](https://arxiv.org/abs/2206.10540) find
  12. Sun, F., Liu, Y., Wang, J.-X., and Sun, H. Symbolic physics learner: Discovering governing equations via monte carlo tree search. In The Eleventh International Conference on Learning Representations, 2023. find
  13. Meidani, K., Shojaee, P., Reddy, C. K., and Farimani, A. B. Snip: Bridging mathematical symbolic and numeric realms with unified pre-training. In The Twelfth International Conference on Learning Representations, 2023.
  14. Udrescu, S.-M. and Tegmark, M. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. doi: 10.1126/sciadv. aay2631.
  15. Uy, N. Q., Hoai, N. X., O’Neill, M., McKay, R. I., and Galván-López, E. Semantically-based crossover in genetic programming: application to real-valued symbolic regression. Genetic Programming and Evolvable Machines, 12:91–119, 2011. find
  16. Merler, M., Haitsiukevich, K., Dainese, N., and Marttinen, P. In-context symbolic regression: Leveraging large language models for function discovery. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 589–606, 2024. find
  17. Virgolin, M. and Pissis, S. P. Symbolic regression is NP-hard. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
  18. Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024.
  19. Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, 2023. find
  20. Mundhenk, T. N., Landajuela, M., Glatt, R., Santiago, C. P., faissol, D., and Petersen, B. K. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Sys… find
  21. Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B. Y., Li, B., Ghazi, B., and Kumar, R. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024.
  22. Petersen, B. K., Larma, M. L., Mundhenk, T. N., Santiago, C. P., Kim, S. K., and Kim, J. T. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021. find
  23. Zhang, Y., Zheng, K., Liu, F., Zhang, Q., and Wang, Z. Autoturb: Using large language models for automatic algebraic model discovery of turbulence closure. arXiv preprint arXiv:2410.10657, 2024. — [arXiv:2410.10657](https://arxiv.org/abs/2410.10657) find
  24. Reddy, C. K. and Shojaee, P. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. arXiv preprint arXiv:2412.11427, 2024. — [arXiv:2412.11427](https://arxiv.org/abs/2412.11427) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shojaee-2025-llm-srbench-new-benchmark/