Elamur Research⌕
☰
home / papers / 2025 / Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents

Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents

Paper · 2025 · Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding et al.

Summary

LLM-based research assistants are good at literature review and idea generation, but they struggle to run experiments carefully. Their prompt-driven workflows are prone to hallucination and do not consistently enforce sound methodology. Curie is a multi-agent framework built to close this gap for computer-science research. An Architect agent drafts experimental plans and interprets findings, while several Technician agents build and run controlled experiments. Between them sits the Experimental Rigor Engine, which has three parts:

  • an intra-agent module that checks each agent's output (for example, plan relevance, setup correctness and execution);
  • an inter-agent module that governs task hand-offs and scheduling;
  • a knowledge module that keeps structured, metadata-rich records of progress.

To evaluate Curie, the authors built a new benchmark of 46 experimental questions spanning four CS domains, drawn from influential papers and popular open-source projects. Curie answered these questions correctly 3.4 times more often than the best competing agent. This suggests that explicit validation and coordination, not prompting alone, are what make automated experimentation trustworthy and reproducible.

Key points

  • Curie's design rests on three principles: methodical procedure, reliability, and interpretability. Each principle is handled by a separate module in the Experimental Rigor Engine.
  • Every agent action follows the same cycle. The inter-agent module intercepts it, the intra-agent module validates it, and the task is then routed to the next suitable agent. This cycle lets the system catch errors at intermediate stages rather than only at the end.
  • The Experimentation Benchmark contains 46 tasks of varying difficulty across four computer-science domains, all based on real research papers and widely used open-source projects.
  • The baselines were OpenHands, a strong coding agent on SWE-Bench, and Microsoft Magentic, a generalist multi-agent system. Curie achieved a 3.4× gain in correct answers over the strongest of them.
  • In a case study revisiting Brown et al. (2024) on how drawing many samples affects LLM reasoning performance, Curie reproduced the original finding, extended it by varying sampling temperature, and identified a methodological limitation. The code is open-source at github.com/Just-Curieous/Curie.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury and Ang Chen. 2025. Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents. arXiv:2502.16069.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.16069 · arXiv 2502.16069

@misc{kon2025curie,
  title = {{Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents}},
  author = {Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury and Ang Chen},
  year = {2025},
  doi = {10.48550/arxiv.2502.16069},
  eprint = {2502.16069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.16069}
}

References

31 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agarwal, S., Laradji, I. H., Charlin, L., and Pal, C. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788, 2024.
  2. Anthropic. Introducing computer use, a new claude 3.5 sonnet, neurips.cc/paper_files/paper/2023/file/ Kitano, H. Nobel turing challenge: creating the engine 58168e8a92994655d6da3939e7cc0918-Paper-Datasets_ for scientific discovery. npj Systems Biology and Appli-and_Benchmarks.pdf. cations, 7(1):29,…
  3. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021a. URL https://arxiv.org/abs/ 2009.03300. ↗
  5. Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., and Zhou, B. Large language models as biomedical hypothesis generators: A comprehensive evaluation, 2024. URL https://arxiv.org/abs/2407.08940. ↗
  6. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv.org/abs/2103.03874. ↗
  7. Hilty, J., Muller, B., Pantin, F., and Leuzinger, S. Plant growth: the what, the how, and the why. New Phytologist, 232(1):25–41, 2021. doi: https://doi.org/10.1111/nph. 17610. URL https://nph.onlinelibrary.wiley. com/doi/abs/10.1111/nph.17610. — [doi:10.1111/nph](https://doi.org/10.1111/nph) ↗
  8. Hofseth, L. J. Getting rigorous with scientific rigor. Carcinogenesis, 39(1):21–25, January 2018. find
  9. Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025.
  10. Sourati, J. and Evans, J. A. Accelerating science with human-aware artificial intelligence. Nature human behaviour, 7(10): 1682–1696, 2023. find
  11. Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., and Yu, K. Scieval: A multi-level large language model evaluation benchmark for scientific research. Proceedings of the AAAI Conference on Artificial Intelligence, 38 (17):19053–19061, Mar. 2024a. doi: 10.1609/aaai.v38i17. 29872. URL…
  12. Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Zhang, C., Wei, C., Li, D., Chen, J., Zhang, J., et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024a.
  13. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representation…
  14. Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is chatgpt good at search? investigating large language models as re-ranking agents, 2024b. URL https://arxiv.org/abs/2304.09542. ↗
  15. Tyser, K., Segev, B., Longhitano, G., Zhang, X.-Y., Meeks, Z., Lee, J., Garg, U., Belsten, N., Shporer, A., Udell, M., Te’eni, D., and Drori, I. Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews, 2024. URL https://arxiv.org/abs/2408.10365. ↗
  16. Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., and Elmqvist, N. The hallmark effect: Supporting provenance and transparent use of large language models in writing with interactive visualization. In Proceedings of the CHI Conference on Human Factors in Computing System… find
  17. Huang, Q., Vora, J., Liang, P., and Leskovec, J. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024. URL https://arxiv.org/abs/2310. 03302. ↗
  18. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
  19. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models, 2024a. URL https://arxiv.org/abs/ 2309.05660. ↗
  20. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2024b. URL https://arxiv.org/ abs/2307.10635. ↗
  21. Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024c.
  22. Jin, M., Yu, Q., Shu, D., Zhao, H., Hua, W., Meng, Y., Zhang, Y., and Du, M. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925, 2024.
  23. Wang, Z. Z., Mao, J., Fried, D., and Neubig, G. Agent workflow memory, 2024d. URL https://arxiv.org/abs/2409. 07429. ↗
  24. Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., and Xu, W. Knowledge conflicts for llms: A survey. arXiv preprint arXiv:2403.08319, 2024.
  25. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793, 2024.
  26. Yuan, J., Yan, X., Shi, B., Chen, T., Ouyang, W., Zhang, B., Bai, L., Qiao, Y., and Zhou, B. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback, 2025. URL https://arxiv.org/abs/2501.03916. ↗
  27. Zhang, L., Zhang, Y., Ren, K., Li, D., and Yang, Y. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks, 2024a. URL https://arxiv.org/ abs/2304.14979. ↗
  28. Zhang, S., Gong, C., Wu, L., Liu, X., and Zhou, M. Automl-gpt: Automatic machine learning with gpt, 2023. URL https://arxiv.org/abs/2305.02499. ↗
  29. Zhang, Y., Chen, X., Jin, B., Wang, S., Ji, S., Wang, W., and Han, J. A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv preprint arXiv:2406.10833, 2024b. find
  30. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  31. Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hypothesis generation with large language models. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pp. 117–139. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.nlp4science-1.10. URL http://dx.doi. org/… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kon-2025-curie-rigorous-automated-scientific/