Elamur Research⌕
☰
home / papers / 2025 / BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology

BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology

Paper · 2025 · Ludovico Mitchener, Jon M Laurent, Alex Andonian et al.

Summary

Bioinformatics may be one of the first fields where AI agents could carry out research autonomously. Yet no broad benchmark existed to test whether LLM-based agents can explore real biological data, run long multi-step analyses and interpret the results. The authors, from FutureHouse and ScienceMachine, built BixBench to fill this gap. PhD-level analysts created "capsules." Each capsule bundles a research hypothesis, heterogeneous input data and a Jupyter notebook that performs the analysis. Claude 3.5 Sonnet drafted candidate questions for each capsule, and human experts then filtered and refined them. The authors also released an agent framework that writes and runs Python, R and bash in notebooks. They used it to test GPT-4o and Claude 3.5 Sonnet. Both models struggled. On open-answer questions, Claude reached about 21% accuracy and GPT-4o about 15%. In multiple-choice form with an abstain option, both scored close to chance. These results show that autonomous bioinformatic analysis is still far off. BixBench gives the field a realistic yardstick for measuring progress.

Key points

  • Dataset size: BixBench has 61 expert-curated analytical capsules and 205 open-answer questions, with 1–7 questions per capsule (about 3.8 on average). It is published on Hugging Face (futurehouse/BixBench), and the evaluation code is on GitHub (Future-House/BixBench).
  • Question creation: Claude 3.5 Sonnet (20,241,022 release) produced eight MCQ drafts per capsule. Experts reviewed these drafts, and an LLM pass, run three times, flagged and removed duplicates.
  • Open-answer results: Claude 3.5 Sonnet scored 21% and GPT-4o scored 15%. The tasks average roughly 4.2 hours of expert work, which is longer than in comparable benchmarks such as BLADE or ScienceAgentBench.
  • Abstain option: When an "insufficient information" choice was offered, accuracy fell near random. Without that choice, scores rose above chance, possibly because models fell back on recall. Majority voting over five trajectories did not change these results meaningfully.
  • Plot ablation: Telling the agents not to produce plots or images had no significant effect on scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksDecision Sciences · 306 worksMedicine · 288 worksHealth Informatics · 227 worksInformation Systems and Management · 96 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Ludovico Mitchener, Jon M Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani and Samuel G Rodriques. 2025. BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology. arXiv:2503.00096.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.00096 · arXiv 2503.00096

@misc{mitchener2025bixbench,
  title = {{BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology}},
  author = {Ludovico Mitchener, Jon M Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani and Samuel G Rodriques},
  year = {2025},
  doi = {10.48550/arxiv.2503.00096},
  eprint = {2503.00096},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.00096}
}

References

3 references, 1 of them with a card of ours. ↗ leads to the work.

  1. Anthropic. Introducing the next generation of claude, March 2024. URL https://www.anthropic.com/ news/claude-3-5-sonnet. Accessed: 2024-07-11. ↗
  2. Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., and Khot, T. SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories, September 2024. URL http://arxiv.org/abs/2409.07440. arXiv:2409.07440 [cs]. ↗
  3. Ultimately, BixBench highlights a fundamental shortcoming in current LLM capabilities for conducting rigorous bioinformatics research, and will thus be a valuable resource to guide development of these capabilities and accelerate discovery across biology. 5.1 5.1.1 Chan, J. S., Chowdhury, N., Jaffe,… — MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mitchener-2025-bixbench-comprehensive-benchmark-llm/