Elamur Research⌕
☰
home / papers / 2025 / BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

Paper · 2025 · Kanishk Gandhi, Michael Y. Li, Lyle Goodyear et al.

Summary

BoxingGym addresses a gap in evaluating LLM-based scientific agents. Earlier work tested experimental design and model discovery separately, and no benchmark examined the full loop of proposing a theory, gathering data and revising the theory. The authors built 10 environments based on real scientific models from fields such as psychology and ecology. Each environment is a generative probabilistic model, so an experiment is simply a sample drawn under the agent's chosen design. Agents interact through a language interface and pursue goals stated in natural language.

The benchmark measures two things:

  • Experiment quality is scored with expected information gain (EIG), a concept from Bayesian optimal experimental design.
  • Discovered models are scored through communication: the scientist agent writes an explanation, and the benchmark checks how well a novice agent predicts outcomes using only that text.

Across open and closed models from 7B to 32B parameters, bigger models and closed-source systems tend to do better. Still, every agent struggles on both tasks, and difficulty varies widely by environment. This positions the benchmark as a clear target for progress in automated scientific reasoning.

Key points

  • The benchmark has 10 environments, such as hyperbolic temporal discounting, population growth, death process, animal growth, moral judgments and emotions. It uses modular WorldEnv / Goal / Agent classes in the style of OpenAI Gym, and the code is at github.com/kanishkg/boxing-gym.
  • The models tested include GPT-4o, Qwen2.5 (7B and 32B) and OpenThinker (7B and 32B). Instruction-tuned Qwen2.5 unexpectedly beat the reasoning-tuned OpenThinker models.
  • GPT-4o had the lowest EIG regret. Qwen-32B sometimes predicted well while still choosing less informative experiments.
  • Box's Apprentice, which adds explicit statistical model fitting to the LLM, gave no consistent gains over plain GPT-4o. It often chose overly simple, for example linear, model forms.
  • In the communication test, the scientist experiments for 10 steps and the novice is always gpt-4o. Explanations from 32B models helped the novice more than those from 7B models. However, novices still did worse than the scientists themselves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

Cite

Cite as Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman and Noah D. Goodman. 2025. BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery. arXiv:2501.01540.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.01540 · arXiv 2501.01540

@misc{gandhi2025boxinggym,
  title = {{BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery}},
  author = {Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman and Noah D. Goodman},
  year = {2025},
  doi = {10.48550/arxiv.2501.01540},
  eprint = {2501.01540},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.01540}
}

References

55 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337–371. PMLR, 2023. find
  2. Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4, 2023. find
  3. Anthropic. Claude 3.7 sonnet. https://www.anthropic.com, 2024. ↗
  4. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.
  5. Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. The moral machine experiment. Nature, 563(7729): 59–64, 2018. find
  6. Josh C. Bongard and Hod Lipson. Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104:9943 – 9948, 2007. find
  7. G. E. P. Box and William G. Hunter. A Useful Method for Model-Building. Technometrics, 4: 301–318, 1962. find
  8. George E. P. Box. Sampling and Bayes’ Inference in Scientific Modelling and Robustness. Journal of the Royal Statistical Society. Series A (General), 143(4):383–430, 1980. ISSN 00359238. find
  9. George EP Box. Science and statistics. Journal of the American Statistical Association, 71 (356):791–799, 1976. find
  10. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. find
  11. Pablo Samuel Castro, Nenad Tomasev, Ankit Anand, Navodita Sharma, Rishika Mohanta, Aparna Dev, Kuba Perlin, Siddhant Jain, Kyle Levin, Noémi Éltető, Will Dabney, Alexander Novikov, Glenn C Turner, Maria K Eckstein, Nathaniel D Daw, Kevin J Miller, and Kimberly L Stachenfeld. Discovering symbolic co… — [doi:10.1101/2025.02.05.636732](https://doi.org/10.1101/2025.02.05.636732) find
  12. Kathryn Chaloner and Isabella Verdinelli. Bayesian Experimental Design: A Review. Statistical Science, 10(3):273 – 304, 1995. doi: 10.1214/ss/1177009939. URL https://doi.org/10. 1214/ss/1177009939. — [doi:10.1214/ss/1177009939](https://doi.org/10.1214/ss/1177009939) find
  13. David Roxbee Cox. Analysis of survival data. Chapman and Hall/CRC, 2018. find
  14. Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020. find
  15. David Duvenaud, James Lloyd, Roger Grosse, Joshua Tenenbaum, and Ghahramani Zoubin. Structure discovery in nonparametric regression through compositional kernel search. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 o… find
  16. Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational bayesian optimal experimental design. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sy… find
  17. Adam Foster, Desi R Ivanova, Ilyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 18–24 Jul 2021. find
  18. Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165, 2023.
  19. Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.03683, 2024.
  20. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025.
  21. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  22. Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024.
  23. Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès, and Jure Leskovec. Automated hypothesis validation with agentic sequential falsifications, 2025.
  24. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  25. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  26. Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083, 2024. find
  27. Michael Y Li, Emily B Fox, and Noah D Goodman. Automated Statistical Model Discovery with Language Models. In International Conference on Machine Learning (ICML), 2024.
  28. Michael Y. Li, Vivek Vajipey, Noah D. Goodman, and Emily B. Fox. Critical: Critic automation with language models, 2024. URL https://arxiv.org/abs/2411.06590. — [arXiv:2411.06590](https://arxiv.org/abs/2411.06590) ↗
  29. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  30. Måns Magnusson, Paul Bürkner, and Aki Vehtari. posteriordb: a set of posteriors for Bayesian inference and probabilistic programming, October 2023. find
  31. John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon. A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955. AI magazine, 27(4):12–12, 1955. find
  32. B. A. McKinney, J. E. Crowe, H. U. Voss, P. S. Crooke, N. Barney, and J. H. Moore. Hybrid grammar-based approach to nonlinear dynamical system identification from biological time series. Phys. Rev. E, 73:021912, Feb 2006. doi: 10.1103/PhysRevE.73.021912. — [doi:10.1103/physreve.73.021912](https://doi.org/10.1103/physreve.73.021912) find
  33. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021. find
  34. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  35. Jay I. Myung, Daniel R. Cavagnaro, and Mark A. Pitt. A tutorial on adaptive design optimization. Journal of Mathematical Psychology, 57(3):53–67, 2013. ISSN 0022-2496. doi: https://doi.org/ 10.1016/j.jmp.2013.05.005. URL https://www.sciencedirect.com/science/article/ pii/S0022249613000503. — [doi:10.1016/j.jmp.2013.05.005](https://doi.org/10.1016/j.jmp.2013.05.005) find
  36. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502.14499, 2025.
  37. Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024.
  38. Desmond C Ong, Jamil Zaki, and Noah D Goodman. Affective cognition: Exploring lay theories of emotion. Cognition, 143:141–162, 2015. find
  39. OpenAI. Hello, GPT-4. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: 2024-06-04. ↗
  40. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022.
  41. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023. find
  42. Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D Goodman. Certified deductive reasoning with language models. arXiv preprint arXiv:2306.04031, 2023.
  43. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Confe… find
  44. Tom Rainforth, Robert Cornish, Hongseok Yang, Andrew Warrington, and Frank Wood. On Nesting Monte Carlo Estimators. International Conference on Machine Learning (ICML), 2018. find
  45. Georg Rasch. Probabilistic models for some intelligence and attainment tests. ERIC, 1993. find
  46. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240, 2022.
  47. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using ood examples. Advances in Neural Information Processing Systems, 36, 2024.
  48. John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, et al. Mastering board games by external and internal planning with language models. arXiv preprint arXiv:2412.12119, 2024.
  49. Ben Shababo, Brooks Paige, Ari Pakman, and Liam Paninski. Bayesian inference and online experimental design for mapping neural microcircuits. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Assoc… find
  50. Omar Shaikh, Valentino Chai, Michele J Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309, 2023. find
  51. Open Thoughts Team. Open Thoughts, January 2025. find
  52. Vito Volterra. Variations and fluctuations of the number of individuals in animal species living together. ICES Journal of Marine Science, 3(1):3–51, 1928. find
  53. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024. find
  54. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  55. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gandhi-2025-boxinggym-benchmarking-progress-automated/