Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines

Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines

Paper · 2026 · Jean Seo, Gibaeg Kim, Kihun Shin et al.

Summary

The paper addresses a gap in medical LLM evaluation. Existing benchmarks mostly check final diagnoses or multiple-choice knowledge, not how well a model gathers patient history before a doctor visit. The authors introduce EPAG, a benchmark and open pipeline for this pre-consultation step.

Clinicians built diagnostic guidelines for 26 diseases that can be identified through conversation alone. Each guideline feature is weighted high or medium. OpenAI o3-mini generated 520 synthetic patient profiles, which clinicians then reviewed. In simulated dialogues, a patient-agent answers multiple-choice questions posed by the doctor-agent under test. The collected History of Present Illness (HPI) is scored in two ways:

  • Directly: HPI units are matched against the guidelines.
  • Indirectly: a separate model diagnoses from the dialogue.

Eleven LLMs and a human clinician baseline were compared. Proprietary models beat the human baseline, while open-source ones, including the medical model Medgemma-4B, fell below it. A small fine-tuned open model surpassed larger systems. Longer dialogues yielded more HPI but not better diagnoses. The dialogue language also changed questioning behaviour. The resource supports safer clinical deployment of LLMs.

Key points

  • Dataset and scoring: 520 profiles (20 per disease) cover 26 diseases, 10 ICD-11 chapters, 10 primary and 22 secondary specialties. Scoring includes unweighted and weighted guideline-match scores, plus Top-1 and Top-k diagnosis accuracy.
  • Pipeline setup: apart from the doctor-agent, every agent (patient, organizer, comparer, diagnostician, evaluator) runs on GPT-4o-mini at temperature 0. Code and data are released at github.com/seemdog/EPAG.
  • Five-turn results: GPT-4.1 led with 4.82 unweighted and 8.12 weighted (tied with Claude-3.7-Sonnet), plus 74.56 Top-1 and 83.81 Top-k. The human baseline scored 4.35, 7.29, 68.24 and 80.65. Qwen2.5-7B and Llama-3.2-3B ranked lowest.
  • Fine-tuning: a Qwen2.5-32B model trained on curated task-specific data outperformed larger, costlier models.
  • Language effect: Korean versus English tests with Qwen2.5 (7B/32B/72B) showed a trade-off. Korean dialogues earned higher HPI scores through broader questioning, while English ones gave better diagnosis accuracy through deeper follow-ups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea56 works · 3,269 citations🎓Yonsei UniversitySeoul · South Korea18 works · 501 citations

Also on the byline: Catholic University of Korea · Severance Hospital.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee and Eunho Yang. 2026. Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.6

@inproceedings{seo2026evaluating,
  title = {{Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines}},
  author = {Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee and Eunho Yang},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.6},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.6}
}

References

42 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul Basit, Khizar Hussain, Muhammad Abdullah Hanif, and Muhammad Shafique. 2024. Medaide: Leveraging large language models for on-premise medical assistance on edge devices. find
  2. Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. 2024. A systematic review of testing and evaluation of healthcare applications of large language models (llms). medRxiv, pages 2024–04. find
  3. Shreya Johri, Jaehwan Jeong, Benjamin A. Tran, Daniel I. Schlessinger, Shannon Wongvibulsin, Leandra A. Barnes, Hong-Yu Zhou, Zhou Ran Cai, et al. 2025. An evaluation framework for conversational reasoning in clinical llms during patient interactions. Nature Medicine. find
  4. Balu Bhasuran, Qiao Jin, Yuzhang Xie, Carl Yang, Karim Hanna, Jennifer Costa, Cindy Shavor, Wenshan Han, Zhiyong Lu, and Zhe He. 2025. Preliminary analysis of the impact of lab results on large language model generated differential diagnoses. npj Digital Medicine, 8(1):166. find
  5. Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. 2024. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. find
  6. Xiaolan Chen, Jiayang Xiang, Shanfu Lu, Yexin Liu, Mingguang He, and Danli Shi. 2024. Evaluating large language models in medical applications: a survey. arXiv preprint arXiv:2405.07468. — [arXiv:2405.07468](https://arxiv.org/abs/2405.07468) find
  7. Mirae Kim, Kyubum Hwang, Hayoung Oh, Min Ah Kim, Chaerim Park, Yehwi Park, and Chungyeon Lee. 2024. MILD bot: Multidisciplinary childhood cancer survivor question-answering bot. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 665–676,… find
  8. Julien Delaunay and Jordi Cusido. 2024. Evaluating the performance of large language models in predicting diagnostics for spanish clinical cases in cardiology. Applied Sciences, 15(1):61. find
  9. Dennis Fast, Lisa C. Adams, Felix Busch, Conor Fallon, Marc Huppertz, Robert Siepmann, Philipp Prucker, Nadine Bayerl, Daniel Truhn, Marcus Makowski, Alexander Löser, and Keno K. Bressem. Autonomous medical evaluation for guideline adherence of large language models. npj Digital Medicine, 7. find
  10. Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, Rodrigo Gameiro, Lizhou Fan, Eugene Park, Tristan Lin, Joonsik Yoon, Wonjin Yoon, Maarten Sap, Yulia Tsvetkov, Paul Liang, Xuhai Xu, Xin Liu, Daniel McDuff, Hyeonhoon Lee, Hae Wo… find
  11. Farieda Gaber, Maqsood Shaik, Fabio Allega, Agnes Julia Bilecz, Felix Busch, Kelsey Goon, Vedran Franke, and Altuna Akalin. 2025. Evaluating large language model workflows in clinical decision support Gleb Kumichev, Pavel Blinov, Yulia Kuzkina, Vasily Goncharov, Galina Zubkova, Nikolai Zenovkin, Ale… find
  12. MANA SAMIEE. General practitioners’ perspectives on llm chatbots for shared decision-making. find
  13. Peter Sarvari and Zaid Al-Fagih. 2025. Rapidly benchmarking large language models for diagnosing comorbid patients: comparative study leveraging the llm-as-a-judge method. JMIRx Med, 6:e67661. find
  14. Xiaoming Shi, Jie Xu, Jinru Ding, Jiali Pang, Sichen Liu, Shuqing Luo, Xingwei Peng, Lu Lu, Haihong Yang, Mingtao Hu, Tong Ruan, and Shaoting Zhang. 2023. Llm-mini-cex: Automatic evaluation of large language model for diagnostic conversation. find
  15. Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. 2024. Kormedmcqa: Multi-choice question answering benchmark for korean healthcare professional licensing examinations. find
  16. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Y… — Towards Expert-Level Medical Question Answering with Large Language Models
  17. Brenna Li, Ofek Gross, Noah Crampton, Mamta Kapoor, Saba Tauseef, Mohit Jain, Khai N Truong, and Alex Mariakakis. 2024. Beyond the waiting room: Patient’s perspectives on the conversational nuances of pre-consultation chatbots. In Proceedings of the 2024 CHI Conference on Human Factors in Computing… find
  18. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge.
  19. Yang Tan, Zhixing Zhang, Mingchen Li, Fei Pan, Hao Duan, Zijie Huang, Hua Deng, Zhuohang Yu, Chen Yang, Guoyang Shen, et al. 2024. Medchatzh: A tuning llm for traditional chinese medicine consultations. Computers in biology and medicine, 172:108290. find
  20. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R Webster, Ewa Dominowska, Juraj Gottwei… find
  21. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930– 1940. find
  22. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou,… find
  23. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering.
  24. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models.
  25. Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. 2024. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology–a recent scoping review. Diagnostic pathology, 19(1):43. find
  26. Han Qian, Bin Dong, Jia-jun Yuan, Fan Yin, Zhao Wang, Hai-ning Wang, Han-song Wang, Dan Tian, Wei-hua Li, Bin Zhang, et al. 2021. Pre-consultation system based on the artificial intelligence has a better diagnostic performance than the physicians in the outpatient department of pediatrics. Frontiers… find
  27. Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, and Andrew G Lee. 2024. Large language model (llm)-driven chatbots for neuro-ophthalmic medical education. Eye, 38(4):639–641. find
  28. Justin T Reese, Leonardo Chimirri, Yasemin Bridges, Daniel Danis, J Harry Caufield, Michael A Gargano, Carlo Kroll, Andrew Schmeder, Fengchen Liu, Kyran Wissink, et al. 2025. Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disea… find
  29. Cai Wang, Qian Chen, Weizi Shao, and Xiaofeng He. 2024. Kemedgpt: Intelligent medical pre-consultation with knowledge-enhanced large language model. In 2024 IEEE International Conference on Medical Artificial Intelligence (MedAI), pages 386–391. find
  30. A Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Wenting Chen, Xiang Li, and Yixuan Yuan. 2025. A survey of llm-based agents in medicine: How far are we from baymax? find
  31. Source of Diagnostic Guidelines • Cardiology: American College of Cardiology 5 , American Heart Association 6 find
  32. Isabella C Wiest, Marie-Elisabeth Leßmann, Fabian Wolf, Dyke Ferber, Marko Van Treeck, Jiefu Zhu, Matthias P Ebert, Christoph Benedikt Westphalen, Martin Wermke, and Jakob Nikolas Kather. 2024. Anonymizing medical documents with local, privacy preserving large language models: The llm-anonymizer. me… find
  33. Oncology: National Comprehensive Cancer Network 7 • Stroke: American Heart Association, American Stroke Association 8 Caleb Winston, Cleah Winston, Claris Winston, and Chloe Winston. Medical question-generation for pre-consultation with llm in-context learning. In GenAI for Health: Potential, Trust… find
  34. Gastroenterology: American Gastroenterolog-ical Association Homepage 12 • HIV/AIDS: U.S. Preventive Services Task Force 13 find
  35. Rui Yang, Ting Tan, Wei Lu, Arun Thirunavukarasu, Daniel Ting, and Nan Liu. 2023b. Large language models in health care: Development, applications, and challenges. Health Care Science, 2. find
  36. Pulmonology: Global Initiative for Chronic Obstructive Lung Disease 14 , Global Initiative for Asthma 15 find
  37. Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. 2023. HuatuoGPT, towards taming language model to be a doctor. In Findings of the Association for Computational Linguistics: EMNLP…
  38. Nephrology: Improving Global Outcomes 16 • Diabetes: American Diabetes Association 17 • General Surgery: American College of Surgeons 18 find
  39. Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. 2024. Llm-based medical assistant personalization with short- and long-term memory coordination. find
  40. Rheumatology: European League Against Rheumatism 19 • Endocrinology: American Association of Clinical Endocrinologists 20 find
  41. Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Chenyu You, Xian Wu, Yefeng Zheng, Lei Clifton, Zheng Li, Jiebo Luo, and David A. Clifton. 2024. A survey of large language models in medicine: Principl… find
  42. Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. 2025. Diagnosisarena: Benchmarking diagnostic reasoning for large language models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seo-2026-evaluating-pre-consultation-ability-llms/