Elamur Research⌕
☰
home / papers / 2024 / BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments

BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments

Paper · 2024 · Yusuf Roohani, Andrew Lee, Qian Huang et al.

Summary

Genome-wide CRISPR screens are expensive. Roughly 19,000 protein-coding genes can be repressed or activated, yet usually only a small fraction produce the phenotype of interest. Closed-loop design tests genes in batches and uses each round's results to choose the next. The standard Bayesian optimization approach has three drawbacks: it needs a custom model trained on scarce data, it relies on a hand-crafted acquisition function, and it ignores published biological knowledge. The authors propose BioDiscoveryAgent, which replaces this machinery with a large language model. In every round, the model receives the research goal and all measurements so far. It then proposes the next batch of genes and explains its reasoning. Optional tools let it search the literature, run code on tabular biological data, and get feedback from a second LLM acting as a critic. The agent was evaluated by hit ratio, a recall-style metric, on six screens, one of them unpublished. The Claude 3.5 Sonnet version outperforms trained Bayesian optimization baselines and can also select gene pairs. The results suggest LLM agents can make experiment planning more effective and easier for biologists to inspect.

Key points

  • Gains over baselines: With Claude 3.5 Sonnet, the agent finds on average 21% more hits than Bayesian optimization baselines across six datasets. This equals about 17 extra relevant genes per dataset after five rounds of 128 genes.
  • Harder setting: When restricted to non-essential genes, the improvement over baselines rises to 46%.
  • Data leakage control: One evaluation dataset is unpublished, so it could not appear in the LLM's training data.
  • New two-gene task: In a combinatorial setting, where pairs of genes are perturbed together, the agent is more than twice as accurate as random selection. The authors say this task had not previously been studied in closed-loop design.
  • Example and code: One example task is a screen for regulators of Interferon-gamma (IFNG) over 18,939 candidate genes. Code is released at github.com/snap-stanford/BioDiscoveryAgent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations🎓University of California, San FranciscoSan Francisco · United States5 works · 9,937 citations

Also on the byline: Gladstone Institutes · Arc Research Institute.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Decision Sciences · 306 worksBiochemistry, Genetics and Molecular Biology · 273 worksMolecular Biology · 270 worksManagement Science and Operations Research · 145 works

Cite

Cite as Yusuf Roohani, Andrew Lee, Qian Huang, Jian Vora, Zachary Steinhart, Kexin Huang, Alexander Marson, Percy Liang and Jure Leskovec. 2024. BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments. arXiv:2405.17631.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.17631 · arXiv 2405.17631

@misc{roohani2024biodiscoveryagent,
  title = {{BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments}},
  author = {Yusuf Roohani, Andrew Lee, Qian Huang, Jian Vora, Zachary Steinhart, Kexin Huang, Alexander Marson, Percy Liang and Jure Leskovec},
  year = {2024},
  doi = {10.48550/arxiv.2405.17631},
  eprint = {2405.17631},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.17631}
}

References

65 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing claude, 2023. introducing-claude. find
  2. URL https://www.anthropic.com/index/Anthropic. Introducing claude 3.5 sonnet, 2024a. URL https://www.anthropic.com/ news/claude-3-5-sonnet. ↗
  3. Anthropic. Introducing the next generation of claude, 2024b. URL https://www.anthropic.com/news/claude-3-family. ↗
  4. AutoGPT. AutoGPT: An experimental open-source attempt to make gpt-4 fully autonomous. https://github.com/Significant-Gravitas/Auto-GPT, 2023. ↗
  5. Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.
  6. Julia Carnevale, Eric Shifrut, Nupura Kale, William A Nyberg, Franziska Blaeschke, Yan Yi Chen, Zhongmei Li, Sagar P Bapat, Morgan E Diolaiti, Patrick O’Leary, et al. Rasa2 ablation in t cells boosts antigen sensitivity and long-term function. Nature, 609(7925):174–182, 2022. find
  7. Yiqun T Chen and James Zou. Genept: A simple but hard-to-beat foundation model for genes and cells built from chatgpt. bioRxiv, 2023. find
  8. Brian Cleary, Le Cong, Anthea Cheung, Eric S Lander, and Aviv Regev. Efficient generation of transcriptomic profiles by random composite measurements. Cell, 171(6):1424–1436, 2017. find
  9. Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empowering biomedical discovery with ai agents. arXiv preprint arXiv:2404.02831, 2024.
  10. Marc Gillespie, Bijay Jassal, Ralf Stephan, Marija Milacic, Karen Rothfels, Andrea Senff-Ribeiro, Johannes Griss, Cristoffer Sevilla, Lisa Matthews, Chuqiao Gong, Chuan Deng, Thawfeek Varusai, Eliot Ragueneau, Yusra Haider, Bruce May, Veronica Shamovsky, Joel Weiser, Timothy Brunson, Nasim Sanati, L… find
  11. Daniel B Goodman, Camillia S Azimi, Kendall Kearns, Alexis Talbot, Kiavash Garakani, Julie Garcia, Nisarg Patel, Byungjin Hwang, David Lee, Emily Park, et al. Pooled screening of car t cells identifies diverse immune signaling domains for next-generation immunotherapies. Science translational medici… find
  12. Max A Horlbeck, Albert Xu, Min Wang, Neal K Bennett, Chong Y Park, Derek Bogdanoff, Britt Adamson, Eric D Chow, Martin Kampmann, Tim R Peterson, et al. Mapping the genetic landscape of human cells. Cell, 174(4):953–967, 2018. find
  13. Wenpin Hou and Zhicheng Ji. Geneturing tests gpt models in genomics. BioRxiv, 2023. find
  14. Wenpin Hou and Zhicheng Ji. Assessing gpt-4 for cell type annotation in single-cell rna-seq analysis. Nature Methods, pp. 1–4, 2024. find
  15. Mengzhou Hu, Sahar Alkhairy, Ingoo Lee, Rudolf T Pillich, Robin Bachelder, Trey Ideker, and Dexter Pratt. Evaluation of large language models for discovery of gene set function. Research Square, 2023. find
  16. Kaixuan Huang, Yuanhao Qu, Henry Cousins, William A Johnson, Di Yin, Mihir Shah, Denny Zhou, Russ Altman, Mengdi Wang, and Le Cong. Crispr-gpt: An llm agent for automated design of gene-editing experiments. arXiv preprint arXiv:2404.18021, 2024.
  17. Kexin Huang, Romain Lopez, Jan-Christian Hutter, Takamasa Kudo, Antonio Rios, and Aviv Regev. Sequential optimal experimental design of perturbation screens guided by multi-modal priors. bioRxiv, pp. 2023–12, 2023a. find
  18. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking large language models as ai research agents. ArXiv, abs/2310.03302, 2023b. URL https://api.semanticscholar.org/CorpusID:263671541. ↗
  19. Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics, 40(2):btae075, 2024.
  20. Michael Kalos, Bruce L Levine, David L Porter, Sharyn Katz, Stephan A Grupp, Adam Bagg, and Carl H June. T cells with chimeric antigen receptors have potent antitumor effects and can establish memory in patients with advanced leukemia. Science translational medicine, 3(95):95ra73–95ra73, 2011. find
  21. Minoru Kanehisa, Miho Furumichi, Mao Tanabe, Yoko Sato, and Kanae Morishima. Kegg: new perspectives on genomes, pathways, diseases and drugs. Nucleic Acid Research, 45,D1 (2017): D353-D361, 2017. find
  22. Ross D King, Kenneth E Whelan, Ffion M Jones, Philip GK Reiser, Christopher H Bryant, Stephen H Muggleton, Douglas B Kell, and Stephen G Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971):247–252, 2004. find
  23. Alexander Lachmann, Denis Torre, Alexandra B. Keenan, Kathleen M. Jagodnik, Hoyjin J. Lee, Lily Wang, Moshe C. Silverstein, and Avi Ma’ayan. Massive mining of publicly available rna-seq data from human and mouse. Nature Communications, 9(1), 1366., 2018. find
  24. Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559, 2023.
  25. Pat Langley, Herbert A. Simon, Gary L. Bradshaw, and Jan M. Zytkow. Scientific discovery: Computational explorations of the creative processes. 1987. URL https://api.semanticscholar.org/CorpusID:61752618. ↗
  26. Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024.
  27. Steven A Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R Banaji. Chatgpt as research scientist: Probing gpt’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proceedings of the National Academy of Sciences, 121(35):e2404328121, 2024. find
  28. Wendell A Lim. The emerging era of cell engineering: Harnessing the modularity of cells to program complex biological function. Science, 378(6622):848–852, 2022. find
  29. Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Chatgpt-powered conversational drug editing using retrieval and domain feedback. arXiv preprint arXiv:2305.18090, 2023. — [arXiv:2305.18090](https://arxiv.org/abs/2305.18090) find
  30. Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. arXiv preprint arXiv:2402.03921, 2024. — [arXiv:2402.03921](https://arxiv.org/abs/2402.03921) find
  31. Clare Lyle, Arash Mehrjou, Pascal Notin, Andrew Jesson, Stefan Bauer, Yarin Gal, and Patrick Schwab. Discobax discovery of optimal intervention sets in genomic experiment design. In International Conference on Machine Learning, pp. 23170–23189. PMLR, 2023. find
  32. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, pp. 1–11, 2024. find
  33. Murad R Mamedov, Shane Vedova, Jacob W Freimer, Avinash Das Sahu, Amrita Ramesh, Maya M Arce, Angelo D Meringa, Mineto Ota, Peixin Amy Chen, Kristina Hanspers, et al. Crispr screens decode cancer cell pathways that trigger γδ t cell detection. Nature, 621(7977):188–195, 2023. find
  34. Arash Mehrjou, Ashkan Soleymani, Andrew Jesson, Pascal Notin, Yarin Gal, Stefan Bauer, and Patrick Schwab. Genedisco: A benchmark for experimental design in drug discovery. In International Conference on Learning Representations, 2021. find
  35. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: a multimodal medical few-shot learner. arXiv preprint arXiv:2307.15189, 2023.
  36. Eva Marie Y Moresco, Xiaohong Li, and Bruce Beutler. Going forward with genetics: recent technological advances and forward genetics in mice. The American journal of pathology, 182(5): 1462–1473, 2013. find
  37. Matthew R Nelson, Hannah Tipney, Jeffery L Painter, Judong Shen, Paola Nicoletti, Yufeng Shen, Aris Floratos, Pak Chung Sham, Mulin Jun Li, Junwen Wang, et al. The support of human genetic evidence for approved drug indications. Nature genetics, 47(8):856–860, 2015. find
  38. OpenAI. Gpt-3.5-turbo, 2023. URL https://platform.openai.com/docs/models/ gpt-3-5-turbo. ↗
  39. OpenAI. Introducing gpt-4o, 2024a. URL https://openai.com/index/hello-gpt-4o/. ↗
  40. OpenAI. Openai o1-mini, 2024b. URL https://openai.com/index/ openai-o1-mini-advancing-cost-efficient-reasoning/. ↗
  41. OpenAI. Introducing openai o1-preview, 2024c. URL https://openai.com/index/ openai-o1-mini-advancing-cost-efficient-reasoning/. ↗
  42. Gilchan Park, Byung-Jun Yoon, Xihaier Luo, Vanessa López-Marrero, Patrick Johnstone, Shinjae Yoo, and Francis J Alexander. Comparative performance evaluation of large language models for extracting molecular interactions and pathway knowledge. arXiv preprint arXiv:2307.08813, 2023a. — [arXiv:2307.08813](https://arxiv.org/abs/2307.08813) find
  43. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023b. find
  44. Laralynne Przybyla and Luke A Gilbert. A new era in functional genomics screens. Nature Reviews Genetics, 23(2):89–103, 2022. find
  45. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, pp. 1–3, 2023.
  46. Yusuf Roohani, Kexin Huang, and Jure Leskovec. Predicting transcriptional outcomes of novel multigene perturbations with gears. Nature Biotechnology, pp. 1–9, 2023. find
  47. Carlos G Sanchez, Christopher M Acker, Audrey Gray, Malini Varadarajan, Cheng Song, Nadire R Cochran, Steven Paula, Alicia Lindeman, Shaojian An, Gregory McAllister, et al. Genome-wide crispr screen identifies protein pathways modulating tau protein levels in neurons. Communications biology, 4(1):73… find
  48. Varuni Sarwal, Viorel Munteanu, Timur Suhodolschi, Dumitru Ciorba, Eleazar Eskin, Wei Wang, and Serghei Mangul. Biollmbench: A comprehensive benchmarking of large language models in bioinformatics. bioRxiv, pp. 2023–12, 2023. find
  49. Moritz Schaefer, Stephan Reichl, Rob ter Horst, Adele M Nicolas, Thomas Krausgruber, Francesco Piras, Peter Stepper, Christoph Bock, and Matthias Samwald. Large language models are universal biomedical simulators. bioRxiv, pp. 2023–06, 2023. find
  50. Samantha G Scharenberg, Wentao Dong, Ali Ghoochani, Kwamina Nyame, Roni Levin-Konigsberg, Aswini R Krishnan, Eshaan S Rawat, Kaitlyn Spees, Michael C Bassik, and Monther Abu-Remaileh. An spns1-dependent lysosomal lipid transport pathway that enables cell survival under choline limitation. Science Ad… find
  51. Timo Schick, Jane Dwivedi-Yu, R Dessı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools (2023). arXiv preprint arXiv:2302.04761, 2023.
  52. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
  53. Ralf Schmidt, Zachary Steinhart, Madeline Layeghi, Jacob W Freimer, Raymund Bueno, Vinh Q Nguyen, Franziska Blaeschke, Chun Jimmie Ye, and Alexander Marson. Crispr activation and interference screens decode stimulation responses in primary human t cells. Science, 375(6580): eabj4008, 2022. find
  54. Korbinian Schneeberger. Using next-generation sequencing to isolate mutant genes from forward genetic screens. Nature Reviews Genetics, 15(10):662–676, 2014. find
  55. Noy Shakked and Whitney Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187–192, 2023. doi: 10.1126/science.adh2586. URL https://doi.org/10.1126/science.adh2586. — [doi:10.1126/science.adh2586](https://doi.org/10.1126/science.adh2586) ↗
  56. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv, abs/2303.11366, 2023.
  57. Herbert A. Simon, Pat Langley, and Gary L. Bradshaw. Scientific discovery as problem solving. Synthese, 47:1–27, 1981. URL https://api.semanticscholar.org/CorpusID: 46985700. ↗
  58. Gary Tom, Stefan P Schmid, Sterling G Baird, Yang Cao, Kourosh Darvish, Han Hao, Stanley Lo, Sergio Pablo-García, Ella M Rajaonson, Marta Skreta, et al. Self-driving laboratories for chemistry and materials science. Chemical Reviews, 124(16):9633–9732, 2024. find
  59. Aviad Tsherniak, Francisca Vazquez, Phil G Montgomery, Barbara A Weir, Gregory Kryukov, Glenn S Cowley, Stanley Gill, William F Harrington, Sasha Pantel, John M Krill-Burger, et al. Defining a cancer dependency map. Cell, 170(3):564–576, 2017. find
  60. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023. find
  61. Joy Y Wang and Jennifer A Doudna. Crispr technology: A decade of genome editing is only the beginning. Science, 379(6629):eadd8643, 2023. find
  62. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023.
  63. Gijs Wobben. pymed - pubmed querying with python, 2020. URL https://github.com/ gijswobben/pymed. Accessed: 03/2024. ↗
  64. Michael Wornow, Alejandro Lozano, Dev Dash, Jenelle Jindal, Kenneth W Mahaffey, and Nigam H Shah. Zero-shot clinical trial patient matching with llms. arXiv preprint arXiv:2402.05125, 2024. — [arXiv:2402.05125](https://arxiv.org/abs/2402.05125) find
  65. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roohani-2024-biodiscoveryagent-ai-agent-designing-genetic/