Elamur Research⌕
☰
home / papers / 2026 / Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation

Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation

Paper · 2026 · Xiangxu Zhang, Lei Li, Yanyun Zhou et al.

Summary

Static medical benchmarks built from public licensing exams may overstate how well LLMs diagnose. Their items can leak into training data, and their tidy textbook vignettes ignore how real patients describe problems: vaguely, in lay terms, and often with their own wrong guesses. The authors introduce DyReMe, a dynamic evaluation framework for diagnosis with two parts.

  • DyGen starts from seed cases and builds fresh consultation-style questions. These embed clinically motivated traps, such as differential diagnoses retrieved via web search, misleading self-diagnoses, distracting history or external explanations, and health rumors. Questions are phrased through randomly sampled patient personas and then checked by a validator-refiner loop.
  • EvalMed scores answers on four dimensions: diagnostic accuracy, veracity (correcting misinformation), helpfulness (evidence and advice) and consistency across repeated runs.

The authors tested 12 LLMs on Chinese questions derived from four public datasets. The generated cases proved more challenging than existing benchmarks, and models struggled especially with misdiagnosis cues and lay phrasing. The work offers a refreshable, contamination-resistant stress test for trustworthy medical AI.

Key points

  • Seed data come from four public datasets: RJUA, DxBench, DDX-Plus and Dxy. All experiments are conducted only in Chinese.
  • DyReMe is compared with the LLM-based dynamic evaluation methods DyVal2 and Self-Evolving, which the authors view as relying mainly on surface-level rewrites.
  • In one case study, GPT-4o scored 43.75 and WiNGPT2-9B scored 25.00. GPT-4o found the correct diagnosis (iron deficiency anemia), whereas WiNGPT2-9B followed the patient's wrong self-diagnosis. However, WiNGPT2-9B did better at refuting the accompanying rumor.
  • To limit evaluator self-bias, DyGen and EvalMed use different LLMs, EvalMed scores with RAG and fixed rules, and a self-recognition check showed no significant effect.
  • The code is released, and the benchmark is planned to be refreshed every six months as versioned snapshots. It is text-only and is not clinically validated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China163 works · 21,032 citations🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksMedicine · 275 worksHealth Informatics · 214 worksHealth Professions · 75 worksHealth Information Management · 56 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Xiangxu Zhang, Lei Li, Yanyun Zhou, Xiao Zhou, Yingying Zhang and Xian Wu. 2026. Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1218

@inproceedings{zhang2026inflated,
  title = {{Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation}},
  author = {Xiangxu Zhang, Lei Li, Yanyun Zhou, Xiao Zhou, Yingying Zhang and Xian Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1218},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1218}
}

References

116 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judith A Arter and Joseph R Jenkins. 1979. Differential diagnosis—prescriptive teaching: A critical appraisal. Review of educational research, 49(4):517–555. find
  2. Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Med-Bench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17709–17717. find
  3. Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. 2024a. CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis. Preprint, arXiv:2407.13301. find
  4. Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2024b. HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs. Preprint, arXiv:2311.09774. — [arXiv:2311.09774](https://arxiv.org/abs/2311.09774) find
  5. Qianyu Chen, Xin Li, Kunnan Geng, and Mingzhong Wang. 2023. Context-Aware Safe Medication Recommendations with Molecular Graph and DDI Graph Embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):7053–7060. find
  6. Cyril Chhun, Fabian M Suchanek, and Chloé Clavel. 2024. Do language models enjoy their own stories? prompting large language models for automatic story evaluation. Transactions of the Association for Computational Linguistics, 12:1122–1142.
  7. Philip Chung, Akshay Swaminathan, Alex J Goodell, Yeasul Kim, S Momsen Reincke, Lichy Han, Ben Deverett, Mohammad Amin Sadeghi, Abdel-Badih Ariss, Marc Ghanem, et al. 2025a. Verifact: Verifying facts in llm-generated clinical text with electronic health records. arXiv preprint arXiv:2501.16672. — [arXiv:2501.16672](https://arxiv.org/abs/2501.16672) find
  8. Philip Chung, Akshay Swaminathan, Alex J Goodell, Yeasul Kim, S Momsen Reincke, Lichy Han, Ben Deverett, Mohammad Amin Sadeghi, Abdel-Badih Ariss, Marc Ghanem, et al. 2025b. Verifact: Verifying facts in llm-generated clinical text with electronic health records. arXiv preprint arXiv:2501.16672. — [arXiv:2501.16672](https://arxiv.org/abs/2501.16672) find
  9. Tim R Davidson, Viacheslav Surkov, Veniamin Veselovsky, Giuseppe Russo, Robert West, and Caglar Gulcehre. 2024. Self-recognition in language models. arXiv preprint arXiv:2407.06946. find
  10. DingXiang. 2022. Tips for responding to five-star reviews. Accessed: 2025-07-26. find
  11. Suhana Bedi and et al. 2025. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. Preprint, arXiv:2505.23802. find
  12. Alphaeus Dmonte, Roland Oruche, Marcos Zampieri, Prasad Calyam, and Isabelle Augenstein. 2024. Claim Verification in the Age of Large Language Models: A Survey. Preprint, arXiv:2408.14317.
  13. Leah T Braun, Laura Zwaan, Jan Kiesewetter, Martin R Fischer, and Ralf Schmidmaier. 2017. Diagnostic errors by medical students: results of a prospective qualitative study. BMC medical education, 17(1):191. find
  14. Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xi, Fei Huang, and Jingren Zhou. 2024. AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator. Preprint, arXiv:2402.09742.
  15. Yilu Fang, Patrick Ryan, and Chunhua Weng. 2024. Knowledge-guided generative artificial intelligence for automated taxonomy learning from drug labels. Journal of the American Medical Informatics Association, 31(9):2065–2075. find
  16. Tengfei He and Xiao Zhou. 2025. Motifgpl: Motif-enhanced graph prototype learning for deciphering urban social segregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28079–28087. find
  17. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  18. Elizabeth Healey, Amelia Li Min Tan, Kristen L. Flint, Jessica L. Ruiz, and Isaac Kohane. 2025. A case study on using a large language model to analyze continuous glucose monitoring data. Scientific Reports, 15(1):1143. find
  19. Tyler B. Forbush, Adi V. Gundlapalli, Miland N. Palmer, Shuying Shen, Brett R. South, Guy Divita, Marjorie Carter, Andrew Redd, Jorie M. Butler, and Matthew Samore. 2013. “Sitting on Pins and Needles”: Characterization of Symptom Descriptions in Clinical Notes”. AMIA Summits on Translational Science… find
  20. Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling Synthetic Data Creation with 1,000,000,000 Personas. Preprint, arXiv:2406.20094.
  21. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
  22. Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations.
  23. Mark L Graber, Nancy Franklin, and Ruthanna Gordon. 2005. Diagnostic error in internal medicine. Archives of internal medicine, 165(13):1493–1499. find
  24. Hanze Guo, Jianxun Lian, and Xiao Zhou. 2026. Why not collaborative filtering in dual view? bridging sparse and dense models. ACM Transactions on Information Systems. find
  25. Hanze Guo, Yijun Ma, and Xiao Zhou. 2025a. Sorex: Towards self-explainable social recommendation with relevant ego-path extraction. ACM Transactions on Information Systems. find
  26. Hanze Guo, Jing Yao, Xiao Zhou, Xiaoyuan Yi, and Xing Xie. 2025b. Counterfactual reasoning for steerable pluralistic value alignment of large language models. arXiv preprint arXiv:2510.18526. — [arXiv:2510.18526](https://arxiv.org/abs/2510.18526) find
  27. Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. 2024. Improving Model Evaluation using SMART Filtering of Benchmark Datasets. Preprint, arXiv:2410.20245. find
  28. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. find
  29. Qian Hong, Cheng Bian, Xiao Zhou, Xiaoyu Li, Yelei Li, and Zijing Zeng. 2025. Lost in time? a meta-learning framework for time-shift-tolerant physiological signal transformation. arXiv preprint arXiv:2511.21500. — [arXiv:2511.21500](https://arxiv.org/abs/2511.21500) find
  30. Qian Hong, Siyuan Chang, and Xiao Zhou. 2026. Wed-net: A weather-effect disentanglement network with causal augmentation for urban flow prediction. arXiv preprint arXiv:2601.22586. — [arXiv:2601.22586](https://arxiv.org/abs/2601.22586) find
  31. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. Advances in Neural Information Processi…
  32. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  33. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  34. Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024. Investigating Data Contamination for Pre-training Language Models. Preprint, arXiv:2401.06059. — [arXiv:2401.06059](https://arxiv.org/abs/2401.06059) find
  35. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  36. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  37. Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Zhuo Ran Cai, Roxana Daneshjou, and Pranav Rajpurkar. Craft-md: A conversational evaluation framework for comprehensive assessment of clinical llms. In AAAI 2024 Spring Symposium on Clinical Foundation Models. find
  38. Ariadna Juarez-Garcia, Tim Stokes, Beth Shaw, Janette Camosso-Stefinovic, and Richard Baker. 2006. The costs of epilepsy misdiagnosis in england and wales. Seizure, 15(8):598–605. find
  39. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  40. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  41. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. MedExQA: Medical Question Answering Benchmark with Multiple Explanations. Preprint, arXiv:2406.06331. find
  42. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  43. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  44. Retno Larasati, Anna De Liddo, and Enrico Motta. 2023. Meaningful Explanation Effect on User’s Trust in an AI Medical System: Designing Explanations for Non-Expert Users. ACM Trans. Interact. Intell. Syst., 13(4):30:1–30:39. find
  45. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Proc…
  46. Jinlin Li, Xintong Li, and Xiao Zhou. 2025a. Fap-cd: Fairness-driven age-friendly community planning via conditional diffusion generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28168–28176. find
  47. Lei Li and Xiao Zhou. 2025. Leave no one behind: Enhancing diversity while maintaining accuracy in social recommendation. In International Conference on Database Systems for Advanced Applications, pages 51–67. Springer. find
  48. Xintong Li, Haoran Zhang, and Xiao Zhou. 2025b. Spatio-temporal hierarchical causal models. arXiv preprint arXiv:2511.20558. — [arXiv:2511.20558](https://arxiv.org/abs/2511.20558) find
  49. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024a. An Open Source Data Contamination Report for Large Language Models. Preprint, arXiv:2310.17589. find
  50. Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024b. An Open-Source Data Contamination Report for Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 528–541, Miami, Florida, USA. Association for Computational Linguistics. find
  51. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  52. Jie Liu, Wenxuan Wang, Zizhan Ma, Guolin Huang, Yihang SU, Kao-Jung Chang, Wenting Chen, Haoliang Li, Linlin Shen, and Michael Lyu. 2024b. Medchain: Bridging the gap between llm agents and clinical practice through interactive sequential benchmarking. arXiv preprint arXiv:2412.01605. — [arXiv:2412.01605](https://arxiv.org/abs/2412.01605) find
  53. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. 2023. Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset. Advances in Neural Information Processing Systems, 36… find
  54. Siqi Luo, Hongyi Qin, Hanlin Li, and Cui Huang. 2024. Can ChatGPT provide health information as physicians do? Preliminary findings from a cross-sectional study of online medical consultation. Information Research an international electronic journal, 29(2):419– 426. find
  55. Shiwei Lyu, Chenfei Chi, Hongbo Cai, Lei Shi, Xiaoyan Yang, Lei Liu, Xiang Chen, Deng Zhao, Zhiqiang Zhang, Xianguo Lyu, Ming Zhang, Fangzhou Li, Xiaowei Ma, Yue Shen, Jinjie Gu, Wei Xue, and Yiran Huang. 2024. RJUA-QA: A Comprehensive QA Dataset for Urology. Preprint, arXiv:2312.09785. — [arXiv:2312.09785](https://arxiv.org/abs/2312.09785) find
  56. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Preprint, arXiv:2305.14251.
  57. Nithesh Naik, B. M. Zeeshan Hameed, Dasharathraj K. Shetty, Dishant Swain, Milap Shah, Rahul Paul, Kaivalya Aggarwal, Sufyan Ibrahim, Vathsala Patil, Komal Smriti, Suyog Shetty, Bhavan Prasad Rai, Piotr Chlosta, and Bhaskar K. Somani. 2022. Legal and Ethical Consideration in Artificial Intelligence… find
  58. Mathieu Nendaz and Arnaud Perrier. 2012. Diagnostic errors and flaws in clinical reasoning: Mechanisms and prevention in practice. Swiss Medical Weekly, 142(4344):w13706–w13706. find
  59. Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, and Eric Horvitz. 2024. MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering. Preprint, arXiv:2406.06573. — [arXiv:2406.06573](https://arxiv.org/abs/2406.06573) find
  60. Geoffrey R. Norman, Sandra D. Monteiro, Jonathan Sherbino, Jonathan S. Ilgen, Henk G. Schmidt, and Silvia Mamede. 2017. The Causes of Errors in Clinical Reasoning: Cognitive Biases, Knowledge Deficits, and Dual Process Thinking. Academic Medicine, 92(1):23. find
  61. OpenAI. 2025a. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. Technical report. ↗
  62. OpenAI. 2025b. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/. Accessed: 2025-07-27. ↗
  63. Daniele Orso, Nicola Federici, Roberto Copetti, Luigi Vetrugno, and Tiziana Bove. 2020. Infodemic and the spread of fake news in the covid-19-era. European Journal of Emergency Medicine, 27(5):327–328. find
  64. Zetian Ouyang, Yishuai Qiu, Linlin Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios. Preprint, arXiv:2410.03502. find
  65. Hao-Ting Pai, Wen-Cheng Chung, Xin-Hong Fang, Yu-Hsin Hsu, and Shu-Ting Huang. 2024. The Explainable Analytics for Exploring Misdiagnoses. In Proceedings of the 2024 8th International Conference on Medical and Health Informatics, ICMHI ’24, pages 238–243, New York, NY, USA. Association for Computing… find
  66. Jiazhen Pan, Bailiang Jian, Paul Hager, Yundi Zhang, Che Liu, Friedrike Jungmann, Hongwei Bran Li, Chenyu You, Junde Wu, Jiayuan Zhu, et al. 2025. Beyond benchmarks: Dynamic, automatic and systematic red-teaming agents for trustworthy medical language models. arXiv preprint arXiv:2508.00923. — [arXiv:2508.00923](https://arxiv.org/abs/2508.00923) find
  67. Ye-Jean Park, Abhinav Pillai, Jiawen Deng, Eddie Guo, Mehul Gupta, Mike Paget, and Christopher Naugler. 2024. Assessing the research landscape and clinical utility of large language models: a scoping review. BMC Medical Informatics and Decision Making, 24(1):72. find
  68. Leili Pourafkari, Arezou Tajlil, Samad Ghaffari, Reza-yat Parvizi, Mohammadreza Chavoshi, Kasra Kolah-douzan, Nasrin Khaki, Raziyeh Parizad, Geoffery G. Hobika, and Nader D. Nader. 2017. The frequency of initial misdiagnosis of acute aortic dissection in the emergency department and its impact on ou… find
  69. Jiaxing Qiu, Dongliang Guo, Papini Natalie, Peace Noelle, Levinson Cheri, and Teague R. Henry. 2025. Ensemble of Large Language Models for Curated Labeling and Rating of Free-text Data. Preprint, arXiv:2501.08413. — [arXiv:2501.08413](https://arxiv.org/abs/2501.08413) find
  70. Winning Health AI Research. 2025. Wingpt2. Accessed: 2025-07-26. find
  71. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201.
  72. C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. find
  73. Tian Shen, Yu Li, and Xi Chen. 2024. A Systematic Review of Online Medical Consultation Research. Healthcare, 12(17):1687. find
  74. Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024. VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation. Preprint, arXiv:2406.19276.
  75. David M. Studdert, Michelle M. Mello, William M. Sage, Catherine M. DesRoches, Jordon Peugh, Kinga Zapert, and Troyen A. Brennan. 2005. Defensive Medicine Among High-Risk Specialist Physicians in a Volatile Malpractice Environment. JAMA, 293(21):2609–2617. find
  76. Xiaoyu Tan, Shaojie Shi, Xihe Qiu, Chao Qu, Zhenting Qi, Yinghui Xu, and Yuan Qi. 2023. Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlessness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Indust… find
  77. Yuqi Tang, Jing Yu, Zichang Su, Kehua Feng, Zhihui Zhu, Libin Wang, Lei Liang, Qiang Zhang, Keyan Ding, and Huajun Chen. 2025. Clindef: A dynamic evaluation framework for large language models in clinical reasoning. arXiv preprint arXiv:2512.23440. find
  78. Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022. DDXPlus: A New Dataset For Automatic Medical Diagnosis. Preprint, arXiv:2205.09148. — [arXiv:2205.09148](https://arxiv.org/abs/2205.09148) find
  79. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  80. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. cohen’s kappa and gwet’s ac1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC medical research methodology, 13(1):61. thisisjeffchen. 2023. ddxplus-parser/ddxplus_parse…
  81. Jiaxin Wu, Yizhou Yu, and Hong-Yu Zhou. 2024. Uncertainty Estimation of Large Language Models in Medical Question Answering. Preprint, arXiv:2407.08662. — [arXiv:2407.08662](https://arxiv.org/abs/2407.08662) find
  82. Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. 2024. Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology – a recent scoping review. Diagnostic Pathology, 19(1):43. find
  83. Rui Wu, Zhaopeng Qiu, Jiacheng Jiang, Guilin Qi, and Xian Wu. 2022. Conditional Generation Net for Medication Recommendation. In Proceedings of the ACM Web Conference 2022, pages 935–945, Virtual Event, Lyon France. ACM. find
  84. VolcEngine. 2025a. doubao-pro-32k system card. [Online; accessed 2025-07-31]. find
  85. VolcEngine. 2025b. Functionality overview of the content connectivity plugin. [Online; accessed 2025-07-27]. find
  86. Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. 2024a. SOTOPIA-$\\pi$: Interactive Learning of Socially Intelligent Language Agents. https://arxiv.org/abs/2403.08715v3. — [arXiv:2403.08715](https://arxiv.org/abs/2403.08715) ↗
  87. Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuanjing Huang, and Zhongyu Wei. 2025. Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3310–3328, Abu Dhabi, UAE. Association for Computational…
  88. Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2024b. CMB: A Comprehensive Medical Benchmark in Chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Assoc…
  89. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. Preprint, arXiv:2411.04368.
  90. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2024. LiveBench: A Challenging, Contamination-Free LLM Benchmark. https:… find
  91. Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, and Kilem L Gwet. 2013. A comparison of Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, and Carolyn Rose. 2024. DocLens: Multi-aspect Fine-grained Medical Text Evaluation. In Proceedin… find
  92. Cheng Xu, Shuhao Guan, Derek Greene, and M.-Tahar Kechadi. 2024. Benchmark Data Contamination of Large Language Models: A Survey. Preprint, arXiv:2406.04244.
  93. Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jianheng Tang, and Liang Lin. 2019. End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis. Preprint, arXiv:1901.10623. — [arXiv:1901.10623](https://arxiv.org/abs/1901.10623) find
  94. Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvári. 2024. To Believe or Not to Believe Your LLM. Preprint, arXiv:2406.02543. find
  95. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  96. Chaoqi Yang, Cao Xiao, Fenglong Ma, Lucas Glass, and Jimeng Sun. 2021. SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinations. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 3735–3741, Montreal, Canada. Internat… find
  97. Yefei Yang, Xiaofei Zhang, and Peter K. C. Lee. 2019. Improving the effectiveness of online healthcare platforms: An empirical study with multi-period patient-doctor consultation data. International Journal of Production Economics, 207:70–80. find
  98. Xixian Yong, Jianxun Lian, Xiaoyuan Yi, Xiao Zhou, and Xing Xie. 2025. Motivebench: How far are we from human-like motivational reasoning in large language models? In Findings of the Association for Computational Linguistics: ACL 2025, pages 20059– 20089. find
  99. Xixian Yong, Peilin Sun, Zihe Wang, and Xiao Zhou. 2026. Intelli-planner: Towards customized urban planning via large language model empowered reinforcement learning. arXiv preprint arXiv:2601.21212. — [arXiv:2601.21212](https://arxiv.org/abs/2601.21212) find
  100. Xixian Yong, Xiao Zhou, Yingying Zhang, Jinlin Li, Yefeng Zheng, and Xian Wu. Think or not? exploring thinking efficiency in large reasoning models via an information-theoretic lens. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  101. Yuechun Yu, Han Ying, Haoan Jin, Wenjian Jiang, Dong Xian, Binghao Wang, Zhou Yang, and Mengyue Wu. 2025. Medkgeval: A knowledge graph-based multi-turn evaluation framework for open-ended patient interactions with clinical llms. arXiv preprint arXiv:2510.12224. — [arXiv:2510.12224](https://arxiv.org/abs/2510.12224) find
  102. John Zarocostas. 2020. How to fight an infodemic. The lancet, 395(10225):676. find
  103. Qing T. Zeng and Tony Tse. 2006. Exploring and developing consumer health vocabularies. Journal of the American Medical Informatics Association: JAMIA, 13(1):24–29. find
  104. Haoran Zhang, Youjin Wang, Yi Duan, Rong Fu, Di-anyu Zhao, Sicheng Fan, Shuaishuai Cao, Wentao Guo, and Xiao Zhou. 2026a. Social-jepa: Emergent geometric isomorphism. arXiv preprint arXiv:2603.02263. — [arXiv:2603.02263](https://arxiv.org/abs/2603.02263) find
  105. Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, et al. 2023. Huatuogpt, towards taming language model to be a doctor. In Findings of the association for computational linguistics: EMNLP 2023, pages 10859–10885.
  106. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation. Pre… find
  107. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint, arXiv:2306.05685.
  108. Xiao Zhou, Zhongxiang Zhao, and Hanze Guo. 2025. Tricolore: Multi-behavior user profiling for enhanced candidate generation in recommender systems. IEEE Transactions on Knowledge and Data Engineering. find
  109. Guangpu Zhu, Xiaoyu Li, Xin Sui, Jie Hawes, Ramy Hussein, Qian Hong, Xiao Zhou, Zijing Zeng, and Yelei Li. 2025a. Hypertension risk screening using long-term photoplethysmogram and ballistocar-diograph measurements from a smartwatch. In 2025 47th Annual International Conference of the IEEE Engineeri… find
  110. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. In The Twelfth International Conference on Learning Representations. find
  111. Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. 2024. Dynamic Evaluation of Large Language Models by Meta Probing Agents. In Forty-First International Conference on Machine Learning.
  112. Yanxu Zhu, Shitong Duan, Xiangxu Zhang, Jitao Sang, Peng Zhang, Tun Lu, Xiao Zhou, Jing Yao, Xiaoyuan Yi, and Xing Xie. 2025b. Mohobench: Assessing honesty of multimodal large language models via unanswerable visual questions. arXiv preprint arXiv:2507.21503. — [arXiv:2507.21503](https://arxiv.org/abs/2507.21503) find
  113. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A Benchmarking Platform for Text Generation Models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 1097–1100, New York, NY, USA. Asso…
  114. Xiangxu Zhang, Lei Li, Xiao Zhou, and Zheng Liu. 2026b. R2med: A benchmark for reasoning-driven medical retrieval. Preprint, arXiv:2505.14558. — [arXiv:2505.14558](https://arxiv.org/abs/2505.14558) find
  115. Xiangxu Zhang, Xiao Zhou, Hongteng Xu, and Jianxun Lian. 2026c. Hypemed: Enhancing medication recommendations with hypergraph-based patient relationships. ACM Transactions on Information Systems. find
  116. Zhehao Zhang, Jiaao Chen, and Diyi Yang. 2024. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Information Processing Systems, 37:135904–135942.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-inflated-excellence-or-true-performance/