Elamur Research⌕
☰
home / papers / 2025 / CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

Paper · 2025 · Peter Jansen, Oyvind Tafjord, Marissa Radensky et al.

Summary

CodeScientist is an open-source system for semi-automated scientific discovery in AI research. It targets two weaknesses of earlier systems. First, they mostly tweak existing codebases or work in narrow design spaces. Second, their outputs are usually judged only by paper-style review, with little scrutiny of the code. CodeScientist treats ideation as a genetic search, with an LLM acting as a mutator over combinations of research papers and a library of short codeblocks for routine operations such as calling an LLM, plotting, or running statistics. Its workflow covers ideation, planning, code generation with reflection and debugging, reporting, and meta-analysis across repeated runs, with optional human input at several stages. The authors ran hundreds of experiments on agents and virtual environments. The system reported 19 candidate discoveries, and 6 of these survived external conference-style review, internal expert code review, and larger-scale reruns. These 6 were judged minimally sound and incrementally novel. The discoveries cover new tasks, agents, metrics, and data rather than only benchmark gains. The work also shows that inspecting code is essential, because LLM-written programs often do something different from what their reports describe.

Key points

  • Ideas come from genetic search over literature and codeblocks. This combines the open-endedness of paper-based ideation with the practicality of conditioning on code the executor can actually run.
  • Of 19 reported discoveries, 6 counted as candidate discoveries. More than half of the 19 were rejected in code review for serious flaws, such as quietly substituting random actions for the intended mechanism.
  • A typical experiment cost about $4 and took about 2 hours. Each idea was run as 5 independent attempts and then meta-analyzed. Small sample sizes mean false positives and missed small effects are both likely.
  • The domain was agents and virtual environments, using tools such as TextWorldExpress and gpt-4o-mini. One example finding is that an LLM's self-reported confidence correlates only weakly with its accuracy on state prediction.
  • Limitations include discoveries that are at best incremental, and manual filtering to the first 50 implementable ideas, which prevents estimating ideator recall. Code: github.com/allenai/codescientist.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld and Peter Clark. 2025. CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. arXiv:2503.22708 · ACL 2025 (Findings).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.22708 · arXiv 2503.22708

@misc{jansen2025codescientist,
  title = {{CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation}},
  author = {Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld and Peter Clark},
  year = {2025},
  doi = {10.48550/arxiv.2503.22708},
  eprint = {2503.22708},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.22708}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. 2023. Symbolic discovery of optimization algorithms. ArXiv, abs/2302.06675. find
  2. Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569–593. Springer. find
  3. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
  4. Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. 2023. Do language models have coherent mental models of everyday things? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1892–1913, Toronto, Canada. Association for Computation… find
  5. Erik Hemberg, Stephen Moskal, and Una-May O’Reilly. 2024. Evolving code with a large language model. Genet. Program. Evolvable Mach., 25:21. find
  6. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning.
  7. John D Hunter. 2007. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(03):90–95. find
  8. Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. 2024. Autonomous llm-driven research from data to human-verifiable research papers. ArXiv, abs/2404.17605.
  9. Intology AI. 2025. Zochi technical report. Technical report, Intology AI. find
  10. Peter Jansen. 2022. A systematic survey of text worlds as embodied natural language environments. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022), pages 1–15, Seattle, United States. Association for Computational Linguistics. find
  11. Peter Jansen and Marc-alexandre Cote. 2023. TextWorldExpress: Simulating text games at one million steps per second. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 169–177, Dubrovnik, Croatia. Association f… find
  12. Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. 2024. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. In The Thirty-eight Conference on Neural…
  13. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  14. John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain,… — Highly accurate protein structure prediction with AlphaFold
  15. Ruochen Li, Teerth Patel, Qingyun Wang, Qingyun Wang, and Xinya Du. 2024. Mlr-copilot: Autonomous machine learning research based on large language models agents. ArXiv, abs/2408.14033.
  16. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology. find
  17. Zijun Liu, Kai Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang, Zhenhe Zhang, Peng Li, and Yang Liu. 2024. Aigs: Generating science from ai-powered automated falsification. ArXiv, abs/2411.11910. — [arXiv:2411.11910](https://arxiv.org/abs/2411.11910) find
  18. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. 2024a. The ai scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408.06292.
  19. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. 2024b. The ai scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408.06292.
  20. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems. find
  21. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  22. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  23. Jingyuan Qi, Zian Jia, Minqian Liu, Wangzhi Zhan, Junkai Zhang, Xiaofei Wen, Jingru Gan, Jianpeng Chen, Qin Liu, Mingyu Derek Ma, Bangzheng Li, Haohui Wang, Adithya Kulkarni, Muhao Chen, Dawei Zhou, Ling Li, Wei Wang, and Lifu Huang. 2024. Metascientist: A human-ai synergistic framework for automate… find
  24. Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S. Weld. 2024. Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination. ArXiv, abs/2409.14634. — [arXiv:2409.14634](https://arxiv.org/abs/2409.14634) find
  25. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using llm agents as research assistants. In arXiv, volume abs/2501.04227.
  26. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109.
  27. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. find
  28. Jonathan M. Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andrés Cubillos-Ruiz, Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Bloom-Ackermann, Victoria M. Tran, Anush Chiappino-Pepe, Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, T. Jaa… find
  29. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023a. Scimon: Scientific inspiration machines optimized for novelty. In Annual Meeting of the Association for Computational Linguistics.
  30. Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. 2024. Can language models serve as text-based world simulators? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–17, Bang… find
  31. Ruoyao Wang, Graham Todd, Xingdi Yuan, Ziang Xiao, Marc-Alexandre Côté, and Peter Jansen. 2023b. ByteSized32: A corpus and challenge task for generating task-specific world models expressed as text games. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page… find
  32. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  33. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? ArXiv, abs/2304.02015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jansen-2025-codescientist-end-end-semi/