Elamur Research⌕
☰
home / papers / 2025 / HealthBench: Evaluating Large Language Models Towards Improved Human Health

HealthBench: Evaluating Large Language Models Towards Improved Human Health

Summary

Most medical benchmarks for LLMs use multiple-choice or short-answer questions. These capture little of how people actually seek health help, are rarely checked against expert judgment, and are often close to saturated. HealthBench, from OpenAI, addresses this with 5,000 realistic single- and multi-turn health dialogues involving lay users and clinicians, across many languages, regions and personas. The model must reply to the final user message. Each dialogue has its own rubric, written by physicians, that awards or deducts points for specific content and behaviors. A model-based grader applies these rubrics, and its verdicts agree with physicians about as often as physicians agree with one another. Scores have risen steeply: from 16% for GPT-3.5 Turbo to 32% for GPT-4o and 60% for o3. Recent models outperform physicians writing responses without help. Two variants are included: a physician-consensus subset and a hard subset that remains far from solved. The benchmark gives the field an open, extensible and unsaturated standard for judging whether models are safe and useful in health settings.

Key points

  • Scale and authorship: 262 physicians from 26 specialties, with practice experience in 60 countries, wrote 48,562 unique rubric criteria. Each criterion carries a weight from −10 to +10, and a typical example has 11 criteria.
  • Structure and scoring: The benchmark is organized into seven themes (e.g., emergency referrals, global health, context seeking) and five axes (e.g., accuracy, completeness, communication, instruction following). Per-example scores are averaged and clipped to [0, 1].
  • Model results: GPT-4.1 nano beats GPT-4o at about 25× lower cost. The weakest areas are worst-case reliability and asking users for missing context.
  • Variants:
    • HealthBench Consensus contains 3,671 examples scored only on 34 physician-validated criteria, which appear 8,053 times in total.
    • HealthBench Hard contains 1,000 difficult examples, and no evaluated model exceeds 32% on it.
  • Physician baselines and release: Physicians could improve responses from September 2024 models but not from April 2025 models. Data and code are available in OpenAI's simple-evals repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.08775 · arXiv 2505.08775

@misc{arora2025healthbench,
  title = {{HealthBench: Evaluating Large Language Models Towards Improved Human Health}},
  author = {Arora, Rahul K. and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Quinonero-Candela, Joaquin and Tsimpourlas, Foivos and Sharman, Michael and Shah, Meghan and Vallone, Andrea and Beutel, Alex and Heidecke, Johannes and Singhal, Karan},
  year = {2025},
  doi = {10.48550/arxiv.2505.08775},
  eprint = {2505.08775},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.08775}
}

References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. W. Ayers, A. Poliak, M. Dredze, E. C. Leas, Z. Zhu, J. B. Kelley, D. J. Faix, A. M. Goodman, C. A. Longhurst, M. Hogarth, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine, 183(6):589–596, 20… find
  2. H. P. Baker, E. Dwyer, S. Kalidoss, and et al. Chatgpt’s ability to assist with clinical documentation: A randomized controlled trial. Journal of the American Academy of Orthopaedic Surgeons, 32(3):123–129, 2024. doi: 10.5435/JAAOS-D-23-00474. — [doi:10.5435/jaaos-d-23-00474](https://doi.org/10.5435/jaaos-d-23-00474) find
  3. A. L. Beam and I. S. Kohane. Big data and machine learning in health care. JAMA, 319(13):1317–1318, 2018. doi: 10.1001/jama.2017.18391. — [doi:10.1001/jama.2017.18391](https://doi.org/10.1001/jama.2017.18391) find
  4. G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669, 2024. find
  5. S. Clémençon, I. Colin, and A. Bellet. Scaling-up empirical risk minimization: Optimization of incomplete u-statistics. Journal of Machine Learning Research, 17(76):1–36, 2016. URL http://jmlr.org/papers/ v17/15-012.html. ↗
  6. J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, et al. The future landscape of large language models in medicine. Communications Medicine, 3:141, 2023. doi: 10.1038/s43856-023-00370-1. — [doi:10.1038/s43856-023-00370-1](https://doi.org/10.1038/s43856-023-00370-1) find
  7. J. Cosentino, A. Belyaeva, X. Liu, N. A. Furlotte, Z. Yang, C. Lee, E. Schenck, Y. Patel, J. Cui, L. D. Schneider, et al. Towards a personal health large language model. arXiv preprint arXiv:2406.06474, 2024. — [arXiv:2406.06474](https://arxiv.org/abs/2406.06474) find
  8. D. Dash, R. Thapa, J. M. Banda, A. Swaminathan, M. Cheatham, M. Kashyap, N. Kotecha, J. H. Chen, S. Gombar, L. Downing, et al. Evaluation of gpt-3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. arXiv preprint arXiv:2304.13714, 2023. find
  9. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115–118, 2017. doi: 10.1038/nature21056. — [doi:10.1038/nature21056](https://doi.org/10.1038/nature21056) find
  10. D. Fast, L. C. Adams, F. Busch, C. Fallon, M. Huppertz, R. Siepmann, P. Prucker, N. Bayerl, D. Truhn, M. Makowski, et al. Autonomous medical evaluation for guideline adherence of large language models. NPJ Digital Medicine, 7(1):1–14, 2024. find
  11. S. L. Fleming, A. Lozano, W. J. Haberkorn, J. A. Jindal, E. Reis, R. Thapa, L. Blankemeier, J. Z. Genkins, E. Steinberg, A. Nayak, et al. Medalign: A clinician-generated dataset for instruction following with electronic medical records. In Proceedings of the AAAI Conference on Artificial Intelligenc… find
  12. A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash, et al. How does chatgpt perform on the united states medical licensing examination (usmle)? the implications of large language models for medical education and knowledge assessment. JMIR medical education, 9(1):e453… find
  13. E. Goh, R. Gallo, J. Hom, E. Strong, Y. Weng, H. Kerman, J. A. Cool, Z. Kanjee, A. S. Parsons, N. Ahuja, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Network Open, 7(10):e2440969–e2440969, 2024. find
  14. V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega, and D. R. Webster. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photo… — [doi:10.1001/jama.2016.17216](https://doi.org/10.1001/jama.2016.17216) find
  15. V. Hartman, X. Zhang, R. Poddar, and et al. Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Network Open, 7(12):e2448723, 2024. doi: 10.1001/ jamanetworkopen.2024.48723. find
  16. M. V. Heinz, N. Jacobson, E. Wright, and et al. Randomized trial of a generative ai chatbot for mental health treatment. NEJM AI, 1(2):e2400802, 2025. doi: 10.1056/AIoa2400802. — [doi:10.1056/aioa2400802](https://doi.org/10.1056/aioa2400802) find
  17. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. find our card
  18. S. Jain, A. Agrawal, A. Saporta, S. Q. Truong, D. N. Duong, T. Bui, P. Chambon, Y. Zhang, M. P. Lungren, A. Y. Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463, 2021. — [arXiv:2106.14463](https://arxiv.org/abs/2106.14463) find
  19. D. Jin et al. What disease does this patient have? a large-scale open-domain medical qa dataset. arXiv preprint arXiv:2010.06081, 2021. — [arXiv:2010.06081](https://arxiv.org/abs/2010.06081) find
  20. Q. Jin, D. Dhingra, and W. Cohen. Pubmedqa: A dataset for biomedical research question answering. In EMNLP, 2019. find our card
  21. Z. Kanjee, B. Crowe, and A. Rodman. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. Jama, 330(1):78–80, 2023. find
  22. P. Lee, S. Bubeck, and J. Petro. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine, 388:1233–1239, 2023. doi: 10.1056/NEJMsr2214184. — [doi:10.1056/nejmsr2214184](https://doi.org/10.1056/nejmsr2214184) find
  23. C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023. find our card
  24. B. Y. Lin, Y. Deng, K. Chandu, A. Ravichander, V. Pyatkin, N. Dziri, R. Le Bras, and Y. Choi. Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In Proceedings of the 13 th International Conference on Learning Representations (ICLR 2025), 2025. URL https://openreview.n… ↗
  25. X. Liu, D. McDuff, G. Kovacs, I. Galatzer-Levy, J. Sunshine, et al. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023. URL https://arxiv.org/abs/2305.15525. — [arXiv:2305.15525](https://arxiv.org/abs/2305.15525) ↗
  26. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022. find
  27. D. McDuff, M. Schaekermann, T. Tu, and et al. Towards accurate differential diagnosis with large language models. Nature, 626:102–118, 2025. doi: 10.1038/s41586-025-08869-4. — [doi:10.1038/s41586-025-08869-4](https://doi.org/10.1038/s41586-025-08869-4) find
  28. M. Moor, L. von Rueden, S. Adler, W. Ping, H. Valentin, et al. Med-flamingo: A multimodal medical few-shot learner. In Proceedings of the 3rd Machine Learning for Health Symposium, pages 353–367. PMLR, 2023. find
  29. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2022… find our card
  30. H. Nori, N. King, S. McKinney, D. Carignan, and E. Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. doi: 10.48550/ARXIV.2303.13375. find our card
  31. J. A. Omiye, H. Gui, S. J. Rezaei, J. Zou, and R. Daneshjou. Large language models in medicine: the potentials and pitfalls: a narrative review. Annals of Internal Medicine, 177(2):210–220, 2024. doi: 10.7326/M23-2772. — [doi:10.7326/m23-2772](https://doi.org/10.7326/m23-2772) find
  32. S. Pal et al. Medmcqa: A large-scale multi-subject multi-choice dataset for the medical domain. In EMNLP, 2022. find
  33. S. R. Pfohl, H. Cole-Lewis, R. Sayres, D. Neal, M. Asiedu, A. Dieng, N. Tomasev, Q. M. Rashid, S. Azizi, N. Rostamzadeh, et al. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, 30(12):3590–3600, 2024. find
  34. K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. find
  35. Scale AI. VISTA: Visual–language understanding leaderboard. https://scale.com/leaderboard/visual_ language_understanding, 2025. Accessed 3 May 2025. ↗
  36. N. Shah, M. Pfeffer, and P. Liang. Holistic evaluation of large language models for medical applications, 2025. URL https://hai.stanford.edu/news/ holistic-evaluation-of-large-language-models-for-medical-applications. ↗
  37. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. find our card
  38. K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8, 2025. find
  39. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. arXiv preprint arXiv:2501.17399, 2025. URL https://arxiv.org/abs/2501.17399. ↗
  40. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. find
  41. S. Takagi, T. Watari, A. Erabi, K. Sakaguchi, et al. Performance of gpt-3.5 and gpt-4 on the japanese medical licensing examination: comparison study. JMIR medical education, 9(1):e48002, 2023. find
  42. R. Tanno, D. G. Barrett, A. Sellergren, S. Ghaisas, S. Dathathri, A. See, J. Welbl, C. Lau, T. Tu, S. Azizi, et al. Collaboration between clinicians and vision–language models in radiology report generation. Nature Medicine, 31(2):599–608, 2025. find
  43. A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. find
  44. E. J. Topol. High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1):44–56, 2019. doi: 10.1038/s41591-018-0300-7. — [doi:10.1038/s41591-018-0300-7](https://doi.org/10.1038/s41591-018-0300-7) find
  45. T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138, 2024. find
  46. T. Tu, M. Schaekermann, A. Palepu, K. Saab, J. Freyberg, R. Tanno, A. Wang, B. Li, M. Amin, Y. Cheng, et al. Towards conversational diagnostic artificial intelligence. Nature, pages 1–9, 2025. find
  47. C. Y. K. Williams, C. R. Subramanian, S. S. Ali, and et al. Physician- and large language model-generated hospital discharge summaries: A blinded, comparative quality and safety study. medRxiv, 2024. doi: 10.1101/2024.09.29.24314562. — [doi:10.1101/2024.09.29.24314562](https://doi.org/10.1101/2024.09.29.24314562) find
  48. J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y. Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024. find our card
  49. C. Zakka, N. Kiani, A. Wong, J. Shi, C. Raffel, et al. Almanac — retrieval-augmented language models for clinical medicine. NEJM AI, 2024. doi: 10.1056/AIoa2300068. — [doi:10.1056/aioa2300068](https://doi.org/10.1056/aioa2300068) find
  50. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2309.11998, 2024. doi: 10.48550/arXiv.2309.11998. find our card
  51. L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2405.01470, 2024. doi: 10.48550/arXiv.2405.01470. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/