Elamur Research⌕
☰
home / papers / 2024 / DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Paper · 2024 · Peter Jansen, Marc-Alexandre Côté, Tushar Khot et al.

Summary

Testing whether AI agents can carry out science from start to finish is hard, because real laboratory experiments are costly and push systems toward narrow, task-specific solutions. The authors address this with DISCOVERYWORLD, a cheap simulated environment. It is text-based, with an optional 2D top-down view. Agents move through a world, handle objects, use instruments and record observations. To succeed, they must come up with hypotheses, design and run experiments, interpret the data and then act on what they learned. The benchmark has 120 tasks drawn from eight themes, ranging from proteomics to rocket science. Scoring is automatic and covers three aspects: whether the task was finished, whether relevant actions were taken, and whether the agent arrived at the correct explanation. Three LLM-based baselines that do well on earlier environments performed poorly here. Human scientists, by contrast, found the tasks difficult but mostly solvable. This gap makes the resource a useful testbed for building general-purpose discovery agents.

Key points

  • Task design: There are 8 themes (Proteomics, Chemistry, Archaeology/radioisotope dating, Reactor Lab, Plant Nutrients, Space Sick, Rocket Science, Translation). Each comes in 3 difficulty levels with parametric variants, giving 120 tasks, plus simpler Unit Test tasks for individual skills.
  • Baseline agents: The three agents were ReAct, Plan+Exec (iterative decomposition) and Hypothesizer, which keeps a CLIN-style memory of hypotheses and measurements.
  • Agent results: ReAct had the highest completion rates, at 38% on easy and 18% on challenge tasks. Hypothesizer was best at recovering explanatory knowledge, but reached only 34% on easy and 8% on challenge tasks. On Unit Tests, agents completed around 60% or more, which suggests they can handle component skills but not full discovery cycles.
  • Human comparison: 11 scientists with an MSc or PhD, recruited through UpWork, averaged 66% task completion and 55% knowledge score.
  • Availability and context: The code is released under Apache-2.0 at github.com/allenai/discoveryworld. The authors present it as harder than ScienceWorld, because tasks demand open-ended ideation and systematic search.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord and Peter Clark. 2024. DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. arXiv:2406.06769.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.06769 · arXiv 2406.06769

@misc{jansen2024discoveryworld,
  title = {{DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents}},
  author = {Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord and Peter Clark},
  year = {2024},
  doi = {10.48550/arxiv.2406.06769},
  eprint = {2406.06769},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.06769}
}

References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models. Nature, 624:570 – 578, 2023. URL https://api.semanticscholar.org/CorpusID:266432059. ↗
  2. A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6:525 – 535, 2023. URL https://api.semanticscholar.org/CorpusID:258059792. ↗
  3. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. find
  4. M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
  5. M.-A. Côté, Á. Kádár, X. Yuan, B. A. Kybartas, T. Barnes, E. Fine, J. Moore, M. J. Hausknecht, L. E. Asri, M. Adada, W. Tay, and A. Trischler. TextWorld: A learning environment for text-based games. In CGW@IJCAI, 2018. URL https://api.semanticscholar.org/ CorpusID:49552345. ↗
  6. L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https:/…
  7. M. J. Hausknecht, P. Ammanabrolu, M.-A. Côté, and X. Yuan. Interactive fiction games: A colossal adventure. In AAAI Conference on Artificial Intelligence, 2019. URL https://api.semanticscholar.org/CorpusID:202565447. ↗
  8. Q. Huang, J. Vora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2024. URL https://api.semanticscholar.org/CorpusID:263671541. ↗
  9. Infocom. Learning zil, 1989. URL https://web.archive.org/web/20100807003406/ http://xlisp.org/zil.pdf. Accessed: 2024-06-05. ↗
  10. J. M. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M.… — Highly accurate protein structure prediction with AlphaFold
  11. T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In ICLR, 2023.
  12. R. D. King, K. E. Whelan, F. M. Jones, P. G. K. Reiser, C. H. Bryant, S. H. Muggleton, D. B. Kell, and S. G. Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427:247–252, 2004. URL https://api.semanticscholar.org/ CorpusID:4428725. ↗
  13. H. Kitano. Artificial intelligence to win the nobel prize and beyond: Creating the engine for scientific discovery. AI magazine, 37(1):39–49, 2016. find
  14. E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. K. Gupta, and A. Farhadi. AI2-THOR: An interactive 3d environment for visual ai. ArXiv, abs/1712.05474, 2017. URL https://api.semanticscholar.org/CorpusID:28328610. ↗
  15. R. Krishna, J. Wang, W. Ahern, P. Sturmfels, P. Venkatesh, I. Kalvet, G. R. Lee, F. S. Morey-Burrows, I. V. Anishchenko, I. R. Humphreys, R. McHugh, D. Vafeados, X. Li, G. A. Sutherland, A. Hitchcock, C. N. Hunter, A. Kang, E. Brackenbrough, A. K. Bera, M. Baek, F. DiMaio, and D. Baker. Generalized… find
  16. H. Küttler, N. Nardelli, A. Miller, R. Raileanu, M. Selvatici, E. Grefenstette, and T. Rocktäschel. The NetHack learning environment. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7671– 7684. Curran Assoc… find
  17. P. Langley. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987. find
  18. B. P. Majumder, B. Dalvi, P. A. Jansen, O. Tafjord, N. Tandon, L. Zhang, C. Callison-Burch, and P. Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. ArXiv, abs/2310.10134, 2023. URL https://api.semanticscholar.org/ CorpusID:264146262. ↗
  19. K. Popper. The logic of scientific discovery. Routledge, 2005. find
  20. A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. ADaPT: As-needed decomposition and planning with language models. In NAACL (Findings), 2023.
  21. Pygame. Pygame. https://www.pygame.org/, 2024. Accessed: 2024-06-05. ↗
  22. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, A. Fawzi, J. Grochow, A. Lodi, J.-B. Mouret, T. Ringer, and T. Yu. Mathematical discoveries from program search with large language models. Nature, 625:468 – 4…
  23. M. Samvelyan, R. Kirk, V. Kurin, J. Parker-Holder, M. Jiang, E. Hambro, F. Petroni, H. Küttler, E. Grefenstette, and T. Rocktäschel. MiniHack the planet: A sandbox for open-ended reinforcement learning research. ArXiv, abs/2109.13202, 2021. URL https://api.semanticscholar.org/CorpusID:237274213. — [arXiv:2109.13202](https://arxiv.org/abs/2109.13202) ↗
  24. A. D. Sendek, E. D. Cubuk, E. R. Antoniuk, G. Cheon, Y. Cui, and E. J. Reed. Machine learning-assisted discovery of solid li-ion conducting materials. Chemistry of Materials, 2018. URL https://api.semanticscholar.org/CorpusID:139501995. ↗
  25. M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL https://arxiv.org/abs/1912.01734. ↗
  26. M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. J. Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. ArXiv, abs/2010.03768, 2020. URL https://api.semanticscholar.org/CorpusID:222208810. ↗
  27. M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. Goulão, A. Kallinteris, A. KG, M. Krimmel, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis. Gymnasium, Mar. 2023. URL https://zenodo.org/record/ 8127025. ↗
  28. J. Wang, M. King, N. Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, F. Song, G. Buttimore, D. Reichert, N. Rabinowitz, L. Matthey, D. Hassabis, A. Lerchner, and M. Botvinick. Alchemy: A structured task distribution for meta-reinforcement learning. arXiv preprint arXiv:210… — [arXiv:2102.02926](https://arxiv.org/abs/2102.02926) find
  29. L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In ACL, 2023. find
  30. R. Wang, P. Jansen, M.-A. Côté, and P. Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022. find
  31. R. Wang, P. A. Jansen, M.-A. Côté, and P. Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://api.semanticscholar.org/CorpusID:247451124. ↗
  32. K. A. Williams, E. Bilsland, A. Sparkes, W. Aubrey, M. Young, L. N. Soldatova, K. D. Grave, J. Ramon, M. de Clare, W. Sirawaraporn, S. G. Oliver, and R. D. King. Cheaper faster drug development validated by the repositioning of drugs against neglected tropical diseases. Journal of the Royal Society… find
  33. M. Xu, G. Jiang, W. Liang, C. Zhang, and Y. Zhu. Interactive visual reasoning under uncertainty, 2023. find
  34. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. InterCode: Standardizing and benchmarking interactive coding with execution feedback. In Advances in Neural Information Processing Systems, volume 36, 2023.
  35. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022. URL https://api.semanticscholar.org/CorpusID:252762395. ↗
  36. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A realistic web environment for building autonomous agents, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jansen-2024-discoveryworld-virtual-environment-developing/