Elamur Research⌕
☰
home / papers / 2026 / Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models

Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models

Paper · 2026 · Wenxuan Wang, Zizhan Ma, Guo Yu et al.

Summary

Many benchmarks now test large language models on medical tasks, but their own quality is rarely checked. Common problems include heavy reliance on multiple-choice exam questions, data drawn from academic sources instead of real clinical records, and little attention to safety. The authors present MedCheck, a framework for auditing medical benchmarks across their whole life. It divides benchmark creation into five phases: design and conceptualization, dataset construction and management, technical implementation and evaluation methodology, validity and performance verification, and documentation, openness and governance. These phases are covered by 46 criteria specific to medicine.

The authors scored 56 existing medical LLM benchmarks with this checklist. Quality varied widely, and three problems appeared across the field:

  • weak links to actual clinical practice;
  • unaddressed risk of training-data contamination;
  • almost no testing of robustness or of how models express uncertainty.

The work matters because inflated or clinically irrelevant scores can create a false sense of progress. MedCheck gives benchmark builders a design guide and gives users a way to vet benchmarks before relying on them.

Key points

  • MedCheck structure: five lifecycle phases and 46 medically tailored criteria. The authors describe it as the first lifecycle-oriented assessment framework for medical benchmarks.
  • Benchmarks audited (56 in total): examples include MedQA, PubmedQA, HealthBench, EHRNoteQA, AgentClinic, MedSafetyBench, PMC-VQA, MedAgentBench, MedS-Bench, GMAI-MMBench and MEDIC.
  • Relation to earlier frameworks: MedCheck builds on general-purpose checklists such as BetterBench (46 criteria) and How2Bench (55 items, for code). It adds medicine-specific concerns, including privacy rules (HIPAA, GDPR), expert involvement, and construct validity.
  • Proposed future directions:
    • interactive, sequential-decision benchmarks (e.g., MediQ, AgentClinic);
    • checking benchmark scores against real-world clinical outcomes, such as EHR-based predictions;
    • a shared "living" repository modeled on betterbench.stanford.edu.
  • Authorship metadata correction: the first author is Wenting Chen (Stanford University). Wenxuan Wang (Renmin University of China) is a co-author, and Zizhan Ma is the corresponding author. The paper appears in ACL 2026 (Long Papers), pp. 43078–43123.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓City University of Hong KongHong Kong · China45 works · 3,014 citations🎓Shenzhen UniversityShenzhen · China16 works · 95 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Wenxuan Wang, Zizhan Ma, Guo Yu, Yiu-Fai Cheung, Meidan Ding, Jie Liu, Wenting Chen and Linlin Shen. 2026. Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1996

@inproceedings{wang2026beyond,
  title = {{Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models}},
  author = {Wenxuan Wang, Zizhan Ma, Guo Yu, Yiu-Fai Cheung, Meidan Ding, Jie Liu, Wenting Chen and Linlin Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1996},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1996}
}

References

87 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. 2025. Medec: A benchmark for medical error detection and correction in clinical notes. Preprint, arXiv:2501.03465. find
  2. Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, Inioluwa Deborah Raji, and Travis Zack. 2025. Medical large language model benchmarks should prioritize construct validity. Preprint, arXiv:2503.10694. — [arXiv:2503.10694](https://arxiv.org/abs/2503.10694) find
  3. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. Preprint…
  4. Mohammad Arvan, Luís Pina, and Natalie Parde. 2022. Reproducibility in computational linguistics: Is source code enough? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10423–10432, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… find
  5. Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M. Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey Jain, Leonardo Schettini, Mehr Kashyap, Jason Alan Fries, Akshay Swaminathan, Philip Chung, Fateme Nateghi, Asad Aali, and 62 others. 2025. Medhelm:… find
  6. Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman, Sadid A Hasan, and Henning Müller. 2021. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. In Proceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-working notes. 2… find
  7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  8. Eshta Bhardwaj, Harshit Gujral, Siyi Wu, Ciara Zogheib, Tegan Maharaj, and Christoph Becker. 2024. Machine learning data practices through a data curation lens: An evaluation framework. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1055–1067. find
  9. Kathrin Blagec, Jakob Kraiger, Wolfgang Frühwirt, and Matthias Samwald. 2023. Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals. Journal of Biomedical Informatics, 137:104274. find
  10. Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, and Shing-Chi Cheung. 2025. How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs. Preprint, a… — [arXiv:2501.10711](https://arxiv.org/abs/2501.10711) find
  11. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. In Proceedings of the 22nd Chinese National Conference…
  12. Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Yu Qiao, and Junjun He. 2024. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards… find
  13. Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. Preprint, arXiv:2502.14425. — [arXiv:2502.14425](https://arxiv.org/abs/2502.14425) find
  14. K. Bretonnel Cohen, Dina Demner-Fushman, rightfully rightfully, and Asma Ben Abacha. 2018. Three dimensions of reproducibility in natural language processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  15. Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests. Psychological bulletin, 52(4):281. find
  16. Wei Dai, Peilin Chen, Malinda Lu, Daniel A Li, Haowen Wei, Hejie Cui, and Paul Pu Liang. 2025. CLIMB: Data foundations for large scale multimodal clinical foundation models. In Forty-second International Conference on Machine Learning. find
  17. Christopher Ifeanyi Eke and Liyana Shuib. 2025. The role of explainability and transparency in fostering trust in ai healthcare systems: a systematic literature review, open issues and potential solutions. Neural Computing and Applications, 37(4):1999–2034. find
  18. Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. 2025a. MedOdyssey: A medical domain benchmark for long context evaluation up to 200K tokens. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 32–56, Albuquerque, New Mexico. Association for… find
  19. Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei, and Fei Huang. 2025b. AI hospital: Benchmarking large language models in a multi-agent medical interaction simulator. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10183–10213, Abu Dhabi,…
  20. Dariush D Farhud and Shaghayegh Zokaei. 2021. Ethical issues of artificial intelligence in medicine and healthcare. Iranian journal of public health, 50(11):i. find
  21. Jason Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojciech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Ott, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang, Alison Callahan, Daniel León Periñán, and 24 others. 202… find
  22. Jack Gallifant, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Caccia-mani, Dina Demner-Fushman, Dmitriy Dligach, Roxana Daneshjou, Chrystinne Fernandes, Lasse Hyldig Hansen, Adam Landman, Lisa Lehmann, Liam G. McCoy, Timothy Miller, Amy Moreno, Nikolaj Munch, David Restre… find
  23. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  24. Yanjun Gao, Dmitriy Dligach, Timothy Miller, John Caskey, Brihat Sharma, Matthew M Churpek, and Majid Afshar. 2023. Dr. bench: Diagnostic reasoning benchmark for clinical natural language processing. Journal of biomedical informatics, 138:104286. find
  25. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050, Bangkok, Thailand. A… find
  26. Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. MedSafetyBench: Evaluating and improving the medical safety of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Industry Track, pages 34… find
  27. Chenlu Guo, Nuo Xu, Yi Chang, and Yuan Wu. 2024. Chbench: A chinese dataset for evaluating health in large language models. Preprint, arXiv:2409.15766. — [arXiv:2409.15766](https://arxiv.org/abs/2409.15766) find
  28. Junqing He, Mingming Fu, and Manshu Tu. 2019. Applying deep matching networks to chinese medical question answering: a study and a dataset. BMC medical informatics and decision making, 19(Suppl 2):52. find
  29. Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) find
  30. Holistic AI. 2024. An overview of data contamination. https://www.holisticai.com/blog/ overview-of-data-contamination. ↗
  31. Yutao Hu, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. OmniMedVQA: A new large-scale comprehensive evaluation benchmark for medical lvlm. Preprint, arXiv:2402.09181. — [arXiv:2402.09181](https://arxiv.org/abs/2402.09181) find
  32. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lu… find
  33. Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. 2025. MedAgentBench: A realistic virtual ehr environment to benchmark medical llm agents. Preprint, arXiv:2501.14654. — [arXiv:2501.14654](https://arxiv.org/abs/2501.14654) find
  34. Di Jin, Eileen Pan, Nassib Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  35. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language proce…
  36. Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. 2023. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10(1):1.
  37. Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. 2024. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint arXiv:2409.07314. — [arXiv:2409.07314](https://arxiv.org/abs/2409.07314) find
  38. Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina S Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad W Safranek, Abid A Anwar, Andrew Zhang, Aidan Gilson, Maxwell B Singer, Amisha Dave, Andrew Taylor, Aidong Zhang, Qingyu Chen, and Zhiyong Lu. 2024. Medcalc-bench: Evaluating large lang… find
  39. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. MedExQA: Medical question answering benchmark with multiple explanations. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 167–181, Bangkok, Thailand. Association for Computational Linguistics. find
  40. Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwanghyun Kim, Jee-won Yang, Seunghyun Won, and Edward Choi. 2024. Ehrnoteqa: An llm benchmark for real-world clinical practice using discharge summaries. In Advances in Neural Information Processing Systems, volume 37, pages 12457… find
  41. Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, and Naeemul Khan. 2024. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In Proceedings of the 2024 Conference on Empirica…
  42. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Sci. Data, 5(1):180251. find
  43. Jing Li, Shangping Zhong, and Kaizhi Chen. 2021. MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8862–8874, Online and Punta Cana, Dominican Republic. Association for Computational… find
  44. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In Advances in Neural Information Processing Systems, volume 37, pages 28858– 2888…
  45. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. find
  46. Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, and David A. Clifton. 2024a. Large language models in the clinic: A… — [arXiv:2405.00716](https://arxiv.org/abs/2405.00716) find
  47. Fenglin Liu, Jinge Wu, Hongjian Zhou, Xiao Gu, So-heila Molaei, Anshul Thakur, Lei Clifton, Honghan Wu, and David A Clifton. 2025a. Riskagent: Autonomous medical ai copilot for generalist risk prediction. arXiv preprint arXiv:2503.03802. — [arXiv:2503.03802](https://arxiv.org/abs/2503.03802) find
  48. Jie Liu, Wenxuan Wang, Zizhan Ma, Guolin Huang, Yihang SU, Kao-Jung Chang, Wenting Chen, Haoliang Li, Linlin Shen, and Michael Lyu. 2024b. Medchain: Bridging the gap between llm agents and clinical practice through interactive sequential benchmarking. Preprint, arXiv:2412.01605. — [arXiv:2412.01605](https://arxiv.org/abs/2412.01605) find
  49. Jie Liu, Wenxuan Wang, Su Yihang, Jingyuan Huang, Yudi Zhang, Cheng-Yi Li, Wenting Chen, Xiaohan Xing, Kao-Jung Chang, Linlin Shen, and Michael R. Lyu. 2025b. Asclepius: A spectrum evaluation benchmark for medical multi-modal large language models. In Proceedings of the 63rd Annual Meeting of the As… find
  50. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, LEI ZHU, and Michael Lingzhi Li. 2023. Benchmarking large language models on cmexam - a comprehensive chinese medical exam dataset. In Advances in Neural Information Processing Systems,… find
  51. Lei Liu, Xiaoyan Yang, Junchi Lei, Yue Shen, Jian Wang, Peng Wei, Zhixuan Chu, Zhan Qin, and Kui Ren. 2024c. A survey on medical large language models: Technology, application, trustworthiness, and future directions. arXiv preprint arXiv:2406.03712. — [arXiv:2406.03712](https://arxiv.org/abs/2406.03712) find
  52. Shengyuan Liu, Boyun Zheng, Wenting Chen, Zhihao Peng, Zhenfei Yin, Jing Shao, Jiancong Hu, and Yixuan Yuan. 2025c. A comprehensive evaluation of multi-modal large language models for endoscopy analysis. Preprint, arXiv:2505.23601. — [arXiv:2505.23601](https://arxiv.org/abs/2505.23601) find
  53. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to extrapolation. arXiv preprint arXiv:2205.02340. — [arXiv:2205.02340](https://arxiv.org/abs/2205.02340) find
  54. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453. find
  55. Tobi Olatunji, Abraham Toluwase Owodunni, Charles Nimo, Jennifer Orisakwe, Henok Biadglign Ademtew, Chris Fourie, Foutse Yuehgoh, Jonas Kemp, Stephen Moore, Mardhiyah Sanni, Emmanuel Ayodele, Irfan Essa, Timothy Faniran, Bonaventure F. P. Dossou, Fola Omofoye, Wendy Kinara, Tassallah Abdullahi, Mich… — AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset
  56. Zetian Ouyang, Yishuai Qiu, Linlin Wang, Gerard De Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. CliMedBench: A large-scale Chinese benchmark for evaluating medical large language models in clinical scenarios. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process… find
  57. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR.
  58. Dimitrios P. Panagoulias, Persephone Papatheo-dosiou, Anastasios P. Palamidas, Mattheos Sanou-dos, Evridiki Tsoureli-Nikita, Maria Virvou, and George A. Tsihrintzis. 2024. Cognet-md, an evaluation framework and dataset for large language model benchmarks in the medical domain. Preprint, arXiv:2405.1… — [arXiv:2405.10893](https://arxiv.org/abs/2405.10893) find
  59. Yoonyoung Park, Gretchen Purcell Jackson, Morgan A Foreman, Daniel Gruen, Jianying Hu, and Amar K Das. 2020. Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA open, 3(3):326–331. find
  60. Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Weike Zhao, Zhuoxia Chen, Hongfei Gu, Chuanjin Peng, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025. Quantifying the reasoning abilities of llms on real-world clinical cases. Preprint, arXiv:2503.04691. — [arXiv:2503.04691](https://arxiv.org/abs/2503.04691) find
  61. Nikita Mehandru, Niloufar Golchini, David Bamman, Travis Zack, Melanie F Molina, and Ahmed Alaa. 2025. Er-reason: A benchmark dataset for llm-based clinical reasoning in the emergency room. Preprint, arXiv:2505.22919. — [arXiv:2505.22919](https://arxiv.org/abs/2505.22919) find
  62. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. 2024. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. In Advances in Neural Information Processing Systems, volume 37, pages 21763–21813. Curran Associates, Inc… find
  63. Francesco Molfese, Simone Balloccu, Gianni Fenu, and Ludovico Marras. 2025. Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering. arXiv preprint arXiv:2503.05113. find
  64. Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. AgentClinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. Preprint, arXiv:2405.07960. — [arXiv:2405.07960](https://arxiv.org/abs/2405.07960) find
  65. Artem Shelmanov, Maxim Panov, Ekaterina Sergeevna Fadeeva, Artem Vazhentsev, Roman Konstantinovich Vashurin, and Timothy Baldwin. 2025. Uncertainty quantification for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 5: Tutorial… find
  66. Ofir Ben Shoham and Nadav Rappoport. 2024. Medconceptsqa: Open source medical concepts qa benchmark. Computers in Biology and Medicine, 182:109089. find
  67. Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Yu Rong, Wenbing Huang, Qifeng Bai, and Tingyang Xu. 2025a. Reasonmed: A 370k multi-agent generated dataset for advancing medical reasoning. Preprint, arXiv:2506.09513.
  68. Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, Jingxiong Li, Xinheng Lyu, Tao Lin, and Lin Yang. 2025b. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. In Computer Vision – EC… find
  69. Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, Arman Cohan, and Mark Gerstein. 2025. MedAgentsBench: Benchmarking thinking models and agent frameworks for complex medical reasoning. Preprint, arXiv:2503.07459. — [arXiv:2503.07459](https://arxiv.org/abs/2503.07459) find
  70. The BMJ. 2024. Central oversight, clinical guidelines, and careful implementation of ai in healthcare. https://www.bmj.com/content/384/bmj. q596/rr-1. ↗
  71. Michael Trimble and Paul Hamilton. 2016. The thinking doctor: clinical decision making in contemporary medicine. Clinical Medicine, 16(4):343–346. find
  72. David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, Florence, Italy. Association for Computational Linguistics.
  73. Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2024a. CMB: A comprehensive medical benchmark in Chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Assoc…
  74. Xidong Wang, Nuo Chen, Junyin Chen, Yidong Wang, Guorui Zhen, Chunxian Zhang, Xiangbo Wu, Yan Hu, Anningzhe Gao, Xiang Wan, Haizhou Li, and Benyou Wang. 2024b. Apollo: A lightweight multilingual medical llm towards democratizing medical ai to 6b people. Preprint, arXiv:2403.03640. — [arXiv:2403.03640](https://arxiv.org/abs/2403.03640) find
  75. Zifeng Wang, Qiao Jin, Jiacheng Lin, Junyi Gao, Jathur-shan Pradeepkumar, Pengcheng Jiang, Benjamin Danek, Zhiyong Lu, and Jimeng Sun. 2025. Trialpanorama: Database and benchmark for systematic review and design of clinical trials. Preprint, arXiv:2505.16097. — [arXiv:2505.16097](https://arxiv.org/abs/2505.16097) find
  76. Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah. 2023. The shaky foundations of large language models and foundation models for electronic health records. npj digital medicine, 6(1):135. find
  77. Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025a. Towards evaluating and building versatile large language models for medicine. npj Digital Medicine, 8(1):58. find
  78. Jiageng Wu, Bowen Gu, Ren Zhou, Kevin Xie, Doug Snyder, Yixing Jiang, Valentina Carducci, Richard Wyss, Rishi J Desai, Emily Alsentzer, Leo Anthony Celi, Adam Rodman, Sebastian Schneeweiss, Jonathan H. Chen, Santiago Romero-Brufau, Kueiyu Joshua Lin, and Jie Yang. 2025b. Bridge: Benchmarking large l… — [arXiv:2504.19467](https://arxiv.org/abs/2504.19467) find
  79. Xian Wu, Yutian Zhao, Yunyan Zhang, Jiageng Wu, Zhihong Zhu, Yingying Zhang, Yi Ouyang, Ziheng Zhang, Huimin Wang, Zhenxi Lin, Jie Yang, Shuang Zhao, and Yefeng Zheng. 2025c. Medjourney: benchmark and evaluation of large language models over patient clinical journey. In Proceedings of the 38th Inter… find
  80. Feng Xie, Jun Zhou, Jin Wee Lee, Mingrui Tan, Siqi Li, Logasan S/O Rajnthern, Marcel Lucas Chee, Bibhas Chakraborty, An-Kwok Ian Wong, Alon Dagan, and 1 others. 2022. Benchmarking emergency department prediction models with machine learning and public electronic health records. Scientific Data, 9(1)… find
  81. Lawrence K. Q. Yan, Qian Niu, Ming Li, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Benji Peng, Ziqian Bi, Pohsun Feng, Keyu Chen, Tianyang Wang, Yunze Wang, Silin Chen, Ming Liu, and Junyu Liu. 2024. Large language model benchmarks in medical tasks. arXiv preprint arXiv:2410.21348. — [arXiv:2410.21348](https://arxiv.org/abs/2410.21348) find
  82. Wenxuan Yang, Weimin Tan, Yuqi Sun, and Bo Yan. 2024. A medical data-effective learning benchmark for highly efficient pre-training of foundation models. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 3499–3508, New York, NY, USA. Association for Computing Machin… find
  83. WonJin Yoon, Shan Chen, Yanjun Gao, Zhanzhan Zhao, Dmitriy Dligach, Danielle S Bitterman, Majid Afshar, and Timothy Miller. 2025. Lcd benchmark: long clinical document benchmark on mortality prediction for language models. Journal of the American Medical Informatics Association, 32(2):285–295. find
  84. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. MMMU: A massive multi-discipline multimodal u…
  85. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Llmeval-med: A real-world clinical benchmark for medical llms with physician validation. Pre… find
  86. Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061–74071. find
  87. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Pmc-vqa: Visual instruction tuning for medical visual question answering. Preprint, arXiv:2305.10415. — [arXiv:2305.10415](https://arxiv.org/abs/2305.10415) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-leaderboard-rethinking-medical-benchmarks/