Elamur Research⌕
☰
home / papers / 2026 / DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models

DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models

Paper · 2026 · Yakun Zhu, Zhongzhen Huang, Linjie Mu et al.

Summary

Popular medical LLM benchmarks such as MedQA and MMLU are nearly saturated, with leading models scoring above 90%. They mostly test exam-style knowledge through multiple-choice questions, not realistic diagnosis. The authors address this with DiagnosisArena, an open-ended benchmark where a model must name the final diagnosis for a detailed patient case. Cases come from clinical case reports in 10 leading journals, including Lancet, NEJM and JAMA. Each report is restructured into case information, physical examination and diagnostic tests. Curation combined several rounds of AI-assisted filtering for sufficient, unambiguous information with review by board-certified physicians. GPT-4o grades each prediction as identical, relevant or irrelevant to the reference diagnosis, and top-1 and top-5 accuracy are computed from five candidate answers. Across 11 frontier models, results are poor: even the best reasoning systems stay below 50%, and most other models fall under 20%. Rewriting the same cases as multiple choice raises scores sharply. This suggests option-based formats hide real gaps in clinical reasoning.

Key points

  • The benchmark contains 1,113 segmented case–diagnosis pairs covering 28 medical specialties. The data and evaluation tools are released at github.com/SPIRAL-MED/DiagnosisArena.
  • Open-ended accuracy is 45.82% for o3-mini, 31.09% for o1 and 17.79% for DeepSeek-R1. Claude-3.5-Sonnet and Qwen2.5-Max score below 20%, while the smaller reasoning model QwQ-32B reaches 25.69%.
  • Reasoning training helps: DeepSeek-R1 gains 13.66 points over its base model, DeepSeek-V3.
  • In the multiple-choice variant, DiagnosisArena-MCQ, o1 rises to 61.90%. Baichuan-M1-14B jumps from under 10% to 58.31%.
  • A leakage check compared accuracy across case publication years, including a pilot of 690 JAMA/NEJM cases from 2022–2024. It found no meaningful contamination effect overall, with a slight 2024 drop for Baichuan-M1 only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

Also on the byline: Capital Medical University · Shanghai Chest Hospital · Beijing Anzhen Hospital.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu and Xiaofan Zhang. 2026. DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.151

@inproceedings{zhu2026diagnosisarena,
  title = {{DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models}},
  author = {Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu and Xiaofan Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.151},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.151}
}

References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. claude-3-5-sonnet. find
  2. Huozhi Zhou Liang Song Mingyu Xu Wei Cheng Xiangrong Zeng Yupeng Zhang Yuqi Huo Zecheng Wang Zhengyun Zhao Bingning Wang, Haizhou Zhao and 1 others. 2025. Baichuan-m1: Pushing the medical capability of large language models. arXiv preprint arXiv:2502.12671.
  3. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2024a. Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060.
  4. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024b. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925. find
  5. Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, and 1 others. 2025. An empirical study on eliciting and improving r1-like reasoning models. arXiv preprint arXiv:2503.04548. — [arXiv:2503.04548](https://arxiv.org/abs/2503.04548) find
  6. Pat Croskerry. 2009. A universal model of diagnostic reasoning. Academic medicine, 84(8):1022–1028. find
  7. Google Deepmind. 2025. Gemini 2.5 pro. find
  8. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  9. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  10. Arthur S Elstein, Lee S Shulman, and Sarah A Sprafka. 1978. Medical problem solving: An analysis of clinical reasoning. Harvard University Press. find
  11. Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519.
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  13. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
  14. Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 2024. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489. find
  15. Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. 2025. O1 replication journey– part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458. — [arXiv:2501.06458](https://arxiv.org/abs/2501.06458) find
  16. Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, and Yu Wang. 2025. Meds Θ3: Towards medical small language models with self-evolved slow thinking. arXiv preprint arXiv:2501.12051. — [arXiv:2501.12051](https://arxiv.org/abs/2501.12051) find
  17. Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. 2024. Learning planning-based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658.
  18. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  19. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  20. Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, and 1 others. 2024. Medcalc-bench: Evaluating large language models for medical calculations. Advances in Neural Information Processing Systems, 37:847… find
  21. Simon A Lee and Timothy Lindsey. 2024. Can large language models abstract medical coded language? arXiv preprint arXiv:2403.10822. — [arXiv:2403.10822](https://arxiv.org/abs/2403.10822) find
  22. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  23. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and 1 others. 2023. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36:52430–52452… find
  24. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, and 1 others. 2025. Towards accurate differential diagnosis with large language models. Nature, pages 1–7. find
  25. Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, and 1 others. 2024. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413. find
  26. Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, and Eric Horvitz. 2024. From medprompt to o1: Exploration of run-time strategies for medical challenge problems and beyond. arXiv preprint arXiv:2411.03590. — [arXiv:2411.03590](https://arxiv.org/abs/2411.03590) find
  27. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  28. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, and 1 others. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
  29. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  30. Qwen Team. 2025b. Qwq-32b: Embracing the power of reinforcement learning. find
  31. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2023a. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935.
  32. Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and 1 others. 2023b. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint arXiv:2308.08833.
  33. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing S…
  34. OpenAI. 2025. Openai o3-mini. find
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  36. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR.
  37. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  38. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and 1 others. 2024. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982. find
  39. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  40. Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Weike Zhao, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025. Quantifying the reasoning abilities of llms on real-world clinical cases. arXiv preprint arXiv:2503.04691. — [arXiv:2503.04691](https://arxiv.org/abs/2503.04691) find
  41. Hongzhou Yu, Tianhao Cheng, Ying Cheng, and Rui Feng. 2025. Finemedlm-o1: Enhancing the medical reasoning ability of llm from supervised fine-tuning to test-time training. arXiv preprint arXiv:2501.09213. — [arXiv:2501.09213](https://arxiv.org/abs/2501.09213) find
  42. OpenAI. 2024a. Hello gpt-4o. find
  43. OpenAI. 2024b. Learning to reason with llms. find
  44. Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. 2024. o1-coder: an o1 replication for coding. arXiv preprint arXiv:2412.00154. zhao zy15. 2024. Rarearena. Website. https://github.com/zhao-zy15/RareArena. — [arXiv:2412.00154](https://arxiv.org/abs/2412.00154) ↗
  45. Yang Zhou, Tan Faith, Yanyu Xu, Sicong Leng, Xinxing Xu, Yong Liu, and Rick Siow Mong Goh. 2024. Benchx: A unified benchmark framework for medical vision-language pretraining on chest x-rays. Advances in Neural Information Processing Systems, 37:6625–6647. find
  46. Yakun Zhu, Shaohang Wei, Xu Wang, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. 2024. Menti: Bridging medical calculator and llm agent with nested tool calling. arXiv preprint arXiv:2410.13610. find
  47. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-diagnosisarena-benchmarking-diagnostic-reasoning-large/