Elamur Research⌕
☰
home / papers / 2026 / Human or LLM as Standardized Patients? A Comparative Study in Medical Education

Human or LLM as Standardized Patients? A Comparative Study in Medical Education

Paper · 2026 · Bingquan Zhang, Xiaoxiao Liu, Yuchi Wang et al.

Summary

Standardized patients (SPs), actors trained to portray clinical cases, are central to medical skills training and OSCEs, but they are costly and hard to scale. LLM-based virtual SPs (VSPs) are a cheaper alternative. However, they tend to reveal case details too early, drift across turns, and have rarely been compared directly with human SPs. The authors propose EasyMED, a multi-agent VSP. It separates recognizing the learner's clinical intent from producing the patient's reply, and a separate agent reviews the finished dialogue against expert case checklists to give feedback. They also build SPBench, a benchmark of real SP–doctor dialogues scored on eight expert-defined criteria, which lets virtual and human SP behavior be compared turn by turn. On SPBench, EasyMED tracks human SP behavior more closely than prior VSPs, especially in case consistency and controlled disclosure. In a four-week controlled study, students trained with EasyMED learned about as well as those trained with human SPs. Novices improved faster early on, and EasyMED sessions were cheaper and more flexible to schedule. The work suggests LLM patients can realistically complement traditional SP programs.

Key points

  • EasyMED has three agents. A Patient Agent produces case-grounded replies conditioned on the recognized intent. An Auxiliary/Intent Recognition Agent maps each question to a standardized clinical intent. An Evaluation Agent compares the whole trajectory with case checklists after the session and flags issues such as missed inquiries.
  • SPBench covers 14 medical specialties and uses eight criteria: QC, CC, CD, RC, LC, LN, CS and PD. These assess how well the patient understands the question (QC), stays faithful to the case (CC), paces the release of information (CD), answers fully (RC), reasons coherently (LC), speaks naturally (LN), remains consistent over the conversation (CS) and behaves like a patient (PD).
  • Learners felt much less anxious with EasyMED than with human SPs (0.5 vs. 3.2, p < .01) and practiced more often in evenings and on weekends.
  • EasyMED sessions had more turns (54 vs. 47) and lasted longer (28m49s vs. 15m17s) than human SP sessions.
  • A session cost $0.725 with EasyMED versus $52.95 with a human SP, roughly a 73-fold saving. The authors note that the study ran at a single institution with a small cohort and used text-only interaction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations🎓Wuhan UniversityWuhan · China35 works · 1,127 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Medicine · 241 worksHealth Informatics · 184 worksPublic Health, Environmental and Occupational Health · 10 worksPhysiology · 29 works

Cite

Cite as Bingquan Zhang, Xiaoxiao Liu, Yuchi Wang, Zhou Lei, Qianqian Xie and Benyou Wang. 2026. Human or LLM as Standardized Patients? A Comparative Study in Medical Education. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1243

@inproceedings{zhang2026human,
  title = {{Human or LLM as Standardized Patients? A Comparative Study in Medical Education}},
  author = {Bingquan Zhang, Xiaoxiao Liu, Yuchi Wang, Zhou Lei, Qianqian Xie and Benyou Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1243},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1243}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Almansoori, Komal Kumar, and Hisham Cholakkal. 2025. Self-evolving multi-agent simulations for realistic clinical interactions. arXiv preprint arXiv:2503.22678. — [arXiv:2503.22678](https://arxiv.org/abs/2503.22678) find
  2. Norman B Berman, Steven J Durning, Martin R Fischer, Soren Huwendiek, and Marc M Triola. 2016. The role for virtual patients in the future of medical education. Academic medicine, 91(9):1217–1222. find
  3. Anna Bodonhelyi, Christian Stegemann-Philipps, Alessandra Sonanini, Lea Herschbach, Marton Szep, Anne Herrmann-Werner, Teresa Festl-Wietek, Enkelejda Kasneci, and Friederike Holderried. 2025. Modeling challenging patient interactions: Llms for medical communication training. arXiv preprint arXiv:250… — [arXiv:2503.22250](https://arxiv.org/abs/2503.22250) find
  4. Hsi-Min Chen, Bao-An Nguyen, Yi-Xiang Yan, and Chyi-Ren Dow. 2020. Analysis of learning behavior in an automated programming assessment environment: A code quality perspective. IEEE access, 8:167341–167354. find
  5. Jennifer Cleland and Steven J Durning. 2022. Researching medical education. John Wiley & Sons. find
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  7. David A Cook and Marc M Triola. 2009. Virtual patients: a critical literature review and proposed next steps. Medical education, 43(4):303–311. find
  8. Zhuoyun Du, Lujie Zheng, Renjun Hu, Yuyang Xu, Xiawei Li, Ying Sun, Wei Chen, Jian Wu, Haolei Cai, and Haohao Ying. 2024. Llms can simulate standardized patients via agent coevolution. arXiv preprint arXiv:2412.11716.
  9. Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. 2023. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312.14890.
  10. Zhenhua Gai, Lianxin Tong, and Quan Ge. 2024. Achieving higher factual accuracy in llama llm with weighted distribution of retrieval-augmented generation. find
  11. Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large language models in education: Vision and opportunities. In 2023 IEEE international conference on big data (BigData), pages 4776–4785. IEEE. find
  12. Christian Grévisse. 2024. Raspatient pi: A low-cost customizable llm-based virtual standardized patient simulator. In International Conference on Applied Informatics, pages 125–137. Springer. find
  13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  14. Ilya Gusev. 2024. Pingpong: A benchmark for role-playing language models with user emulation and multi-model evaluation. arXiv preprint arXiv:2409.06820. — [arXiv:2409.06820](https://arxiv.org/abs/2409.06820) find
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR).
  16. Yann Hicke, Jadon Geathers, Kellen Vu, Justin Sewell, Claire Cardie, Jaideep Talwalkar, Dennis Shung, Anyanate Gwendolyne Jack, Susannah Cornes, Mackenzi Preston, and 1 others. 2025. Medsimai: simulation and formative feedback generation to enhance deliberate practice in medical education. arXiv pre… — [arXiv:2503.05793](https://arxiv.org/abs/2503.05793) find
  17. Grace Huang, Robby Reynolds, and Chris Candler. 2007. Virtual patient simulation at us and canadian medical schools. Academic medicine, 82(5):446– 451. find
  18. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  19. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  20. Andrzej A Kononowicz, Nabil Zary, Samuel Edelbring, Janet Corral, and Inga Hege. 2015. Virtual patients-what are we talking about? a framework to classify the meanings of the term in healthcare education. BMC medical education, 15:1–7. find
  21. Junbok Lee, Sungkyung Park, Jaeyong Shin, and Belong Cho. 2024. Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Medical Informatics and Decision Making, 24(1):366. find
  22. Keyeun Lee, Seolhee Lee, Esther Hehsun Kim, Yena Ko, Jinsu Eun, Dahee Kim, Hyewon Cho, Haiyi Zhu, Robert E Kraut, Eunyoung E Suh, and 1 others. 2025. Adaptive-vp: A framework for llm-based virtual patients that adapts to trainees’ dialogue to facilitate nurse communication training. In Findings of t… find
  23. Jiarui Li, Ye Yuan, and Zehua Zhang. 2024a. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv preprint arXiv:2403.10446. — [arXiv:2403.10446](https://arxiv.org/abs/2403.10446) find
  24. Yanzeng Li, Cheng Zeng, Jialun Zhong, Ruoyu Zhang, Minhao Zhang, and Lei Zou. 2024b. Leveraging large language model as simulated patients for clinical education. arXiv preprint arXiv:2404.13066. — [arXiv:2404.13066](https://arxiv.org/abs/2404.13066) find
  25. Ruoyu Liu, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. 2025. Interactive evaluation for medical llms via task-oriented dialogue system. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4871–4896. find
  26. Jinkyoung Ma, Youngjin Lee, and Jiwon Kang. 2023. Standardized patient simulation for more effective undergraduate nursing education: a systematic review and meta-analysis. Clinical Simulation in Nursing, 74:19–37. find
  27. Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluís Màrquez. 2024. Factual confidence of llms: On reliability and robustness of current estimators. arXiv preprint arXiv:2406.13415. find
  28. Ian Steenstra, Farnaz Nouraei, and Timothy Bickmore. 2025. Scaffolding empathy: Training counselors with simulated patients and utterance-level performance visualizations. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–22. find
  29. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930– 1940. find
  30. Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, and Andrew G Lee. 2024. Large language model (llm)-driven chatbots for neuro-ophthalmic medical education. Eye, 38(4):639–641. find
  31. Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. 2024. Large language models for education: A survey and outlook. arXiv preprint arXiv:2403.18105. find
  32. Hanyi Xu, Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Philip S Yu. 2024. Large language models for education: A survey. arXiv preprint arXiv:2405.13001. — [arXiv:2405.13001](https://arxiv.org/abs/2405.13001) find
  33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  34. Jiarui Ye and Hao Tang. 2025. Multimodal large language models for medicine: A comprehensive survey. arXiv preprint arXiv:2504.21051. — [arXiv:2504.21051](https://arxiv.org/abs/2504.21051) find
  35. R Parvathy, MG Thushara, and Jinesh M Kannimoola. 2025. Automated code assessment and feedback: A comprehensive model for improved programming education. IEEE Access. find
  36. Benjamin Zendejas, Amy T Wang, Ryan Brydges, Stanley J Hamstra, and David A Cook. 2013. Cost: the missing outcome in simulation-based medical education research: a systematic review. Surgery, 153(2):160–176. find
  37. Conrad W Safranek, Anne Elizabeth Sidamon-Eristoff, Aidan Gilson, and David Chartash. 2023. The role of large language models in medical education: applications and implications. find
  38. Jialing Zhang, Lingfeng Zhou, Jin Gao, Mohan Jiang, and Dequan Wang. Personaeval: Benchmarking llms on role-playing evaluation tasks. Eric W Sayers, Jeff Beck, Evan E Bolton, J Rodney Brister, Jessica Chan, Donald C Comeau, Ryan Connor, Michael DiCuccio, Catherine M Farrell, Michael Feldgarden, and… find
  39. Simzine. 2025. Standardized vs. virtual patients in medical education. [Accessed: 2025-06-01]. find
  40. Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. 2025. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-human-or-llm-as-standardized/