Elamur Research⌕
☰
home / papers / 2025 / AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

Paper · 2025 · Zhihao Fan, Lai Wei, Jialong Tang et al.

Summary

Medical QA benchmarks give a model the whole case at once. Real diagnosis is different: a clinician must gather facts step by step through conversation and tests. To measure this skill, the authors built AI Hospital, a multi-agent simulator. An LLM under test plays the Doctor, while GPT-3.5 drives two non-player characters: a Patient and an Examiner who returns test results. Both draw their information from real medical records. The Doctor has at most 10 turns to question the Patient, order examinations, and write a final report covering diagnosis and treatment.

On top of the simulator they built the MVME benchmark from expert-screened Chinese medical records. Reports are scored with three methods: automatic matching, GPT-4 judging, and human review. The authors also tested a dispute-resolution scheme in which several Doctor agents discuss their disagreements under a coordinating Center Agent.

All tested models, GPT-4 included, scored far below GPT-4 given the complete record in a single step. The collaboration scheme narrowed this gap only partly. The work shows that interactive information gathering remains a major weakness of LLMs in clinical settings.

Key points

  • Agent roles: AI Hospital has one player agent (Doctor) and two NPC agents (Patient, Examiner). Interactions are capped at 10 turns. Code, data and results are public at github.com/LibertFan/AI_Hospital.
  • Benchmark: MVME contains 506 real-world Chinese medical records. It scores symptom collection, examination recommendation, and diagnosis/treatment.
  • Performance gap: In interactive mode, GPT-4 and other LLMs reach less than 50% of one-step GPT-4's scores on diagnostic accuracy, reasoning and treatment planning. Smaller models do worse, and results differ across medical departments.
  • Failure analysis: Doctors reviewed 219 cases that GPT-4 diagnosed incorrectly. They found three main error types:
    • skipping needed examinations;
    • not asking about key symptoms;
    • insufficient medical knowledge, even when all data was available.
  • Evaluator reliability: GPT-4 ratings agree closely with human ratings. Other LLMs used as judges showed biases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMedicine · 241 worksHealth Informatics · 184 worksHealth Professions · 69 worksHealth Information Management · 50 works

Cite

Cite as Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei and Fei Huang. 2025. AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fan2025ai,
  title = {{AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator}},
  author = {Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei and Fei Huang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.680/}
}

References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ryan Antel, Samira Abbasgholizadeh-Rahimi, Elena Guadagno, Jason M Harley, and Dan Poenaru. 2022. The use of artificial intelligence and virtual reality in Yirong Chen, Zhenyu Wang, Xiaofen Xing, Huimin Zheng, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, and Xiangmin Xu. 2023f.… find
  2. Nigel Collier and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at jnlpba. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73–78.
  3. Peter Croft, Douglas G Altman, Jonathan J Deeks, Kate M Dunn, Alastair D Hay, Harry Hemingway, Linda LeResche, George Peat, Pablo Perel, Steffen E Petersen, et al. 2015. The science of clinical practice: disease diagnosis or patient prognosis? evidence about “what is likely to happen” should shape c… find
  4. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  5. Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569–593. Springer. find
  6. Qizheng Gu, Cong Nie, Ruixiang Zou, Wei Chen, Chaojun Zheng, Dongqing Zhu, Xiaojun Mao, Zhongyu Wei, and Dong Tian. 2020. Automatic generation of electromyogram diagnosis report. In 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1645–1650. IEEE. find
  7. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  8. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  9. Benjamin W Lamb, Helen WL Wong, Charles Vincent, James SA Green, and Nick Sevdalis. 2011. Teamwork and team performance in multidisciplinary cancer teams: development and evaluation of an observational assessment tool. BMJ quality & safety, 20(10):849–856. find
  10. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. find
  11. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1):50–70. find
  12. Jialin Liu, Changyu Wang, and Siru Liu. 2023. Utility of chatgpt in clinical practice. Journal of Medical Internet Research, 25:e48568. find
  13. Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. Biored: a rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23(5):bbac282. find
  14. KJ O’leary, CD Ritter, H Wheeler, MK Szekendi, TS Brinton, and MV Williams. 2010. Teamwork on inpatient medical units: assessing attitudes and barriers. BMJ Quality & Safety, 19(2):117–121. find
  15. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  16. ArXiv, Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248–260. PMLR.
  17. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. find
  18. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36.
  19. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138.
  20. Sajad Sotudeh, Nazli Goharian, and Zachary Young. 2022. Mentsum: A resource for exploring summarization of mental health online posts. arXiv preprint arXiv:2206.00856. find
  21. Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. 2023. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.10537.
  22. PA Trott. 1977. International classification of diseases for oncology. Journal of clinical pathology, 30(8):782. find
  23. Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023a. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975.
  24. Haochun Wang, Chi Liu, Sendong Zhao, Bing Qin, and Ting Liu. 2023b. Chatglm-med. https://github.com/SCIR-HI/Med-ChatGLM. ↗
  25. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023c. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432.
  26. Penni I Watts, Donna S McDermott, Guillaume Alin-ier, Matthew Charnetski, Jocelyn Ludlow, Elizabeth Horsley, Colleen Meakim, and Pooja A Nawathe. 2021. Healthcare simulation standards of best prac-ticetm simulation design. Clinical Simulation in Nursing, 58:14–21. find
  27. Wei Zhu and Xiaoling Wang. 2023. Chatmed: A chinese medical large language model. https://github.com/michael-wzhu/ChatMed. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fan-2025-ai-hospital-benchmarking-large-language/