Elamur Research⌕
☰
home / papers / 2025 / Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning

Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning

Paper · 2025 · Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu et al.

Summary

LLMs score at expert level on medical exam benchmarks, but they work poorly as conversational assistants. They often diagnose from the first message without asking follow-up questions. Their reasoning is not tied to trusted clinical sources, and their tone tends to be impersonal. The authors present APP (Ask Patients with Patience), a multi-turn assistant called Dr.APP that addresses these gaps.

At each turn, the system does the following:

  • maps the dialogue onto symptom entries from the MSD Manual using retrieval;
  • estimates a probability distribution over candidate diseases;
  • draws possible follow-up questions from the guideline's diagnostic sections;
  • simulates likely patient answers and picks the question that minimizes expected entropy (Bayesian active learning).

Its replies use plain language and an empathetic tone. For evaluation, the authors built a benchmark in which LLM patient agents follow profiles taken from real consultations. Clinicians and lay participants then rated the dialogues. APP beat one-shot and multi-turn LLM baselines on diagnostic accuracy and reduced uncertainty faster. It also scored far higher on human-centric qualities. The work shows how guideline grounding and transparent probabilistic questioning can make medical chatbots more trustworthy.

Key points

  • Grounding uses both the professional and consumer editions of the MSD Manual. Symptom matching is done with RAG, and the system shows patients an evolving list of disease probabilities rather than an unranked set of possible causes.
  • The benchmark's patient agents are built from over 300 real doctor–patient interviews from ReMeDi (Yan et al., 2022).
  • Human-centric ratings (5-point scale) for Dr.APP:
    • accessibility 4.54, versus 3.48 for the original ReMeDi dialogues;
    • empathy 4.60, versus 2.78;
    • response relevance 4.48, about 0.7 above the runner-up, Claude-3;
    • relationship fostering 4.54.
  • Question selection is greedy, minimizing entropy one step ahead, so it can settle in local optima. The authors suggest RL or multi-step planning as future fixes.
  • The evaluation relies on simulated patients and transcript review. Real clinical trials are still needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksDialogue systems · 624 works

Cite

Cite as Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu, Fenglin Liu and Junde Wu. 2025. Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.142

@inproceedings{zhu2025ask,
  title = {{Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning}},
  author = {Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu, Fenglin Liu and Junde Wu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.142},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.142}
}

References

31 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. 2023. Disc-medllm: Bridging general large language models and real-world medical consultation. arXiv preprint arXiv:2308.14346. — [arXiv:2308.14346](https://arxiv.org/abs/2308.14346) find
  3. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  4. Felix Busch, Lena Hoffmann, Christopher Rueger, Elon HC van Dijk, Rawen Kader, Esteban Ortiz-Prado, Marcus R Makowski, Luca Saba, Martin Hadamitzky, Jakob Nikolas Kather, et al. 2025. Current applications and challenges in large language models for patient care: a systematic review. Communications M… find
  5. Wei Chen, Shiqi Wei, Zhongyu Wei, and Xuanjing Huang. 2023a. Knse: A knowledge-aware natural language inference framework for dialogue symptom status recognition. arXiv preprint arXiv:2305.16833. find
  6. Yirong Chen, Zhenyu Wang, Xiaofen Xing, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, Xiangmin Xu, et al. 2023b. Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt. arXiv preprint arXiv:2310.15896. — [arXiv:2310.15896](https://arxiv.org/abs/2310.15896) find
  7. Gaurav Kumar Gupta, Aditi Singh, Sijo Valayakkad Manikandan, and Abul Ehtesham. 2025. Digital diagnostics: The potential of large language models in recognizing symptoms of common illnesses. AI, 6(1):13. find
  8. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  9. Zahir Kanjee, Byron Crowe, and Adam Rodman. 2023. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. Jama, 330(1):78– 80. find
  10. Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S Ilgen, Yulia Tsvetkov, and Maarten Sap. 2025a. Aligning llms to ask good questions a case study in clinical reasoning. arXiv preprint arXiv:2502.14860. — [arXiv:2502.14860](https://arxiv.org/abs/2502.14860) find
  11. Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. 2024. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. Advances in Neural Information Processing Systems, 37:28858–28888.
  12. Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2025b. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717. — [arXiv:2504.04717](https://arxiv.org/abs/2504.04717) find
  13. Chihung Lin and Chang-Fu Kuo. 2025. Roles and potential of large language models in healthcare: A comprehensive review. Biomedical Journal, page 100868. find
  14. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. 2025. Towards accurate differential diagnosis with large language models. Nature, pages 1–7. find
  15. MSD Manual. 2025a. Msd manual consumer version. find
  16. MSD Manual. 2025b. Msd manual professional edition. find
  17. Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E Priebe, and Eric Horvitz. 2024. Medfuzz: Exploring the robustness of large language models in medical question answering. arXiv preprint arXiv:2406.06573. — [arXiv:2406.06573](https://arxiv.org/abs/2406.06573) find
  18. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452. find
  19. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  20. Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. 2025. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634. — [arXiv:2502.19634](https://arxiv.org/abs/2502.19634) find
  21. Sara Pieri, Sahal Shaji Mullappilly, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Timothy Baldwin, and Hisham Cholakkal. 2024. Bimedix: Bilingual medical mixture of experts llm. arXiv preprint arXiv:2402.13253. find
  22. Alec Radford. 2018. Improving language understanding by generative pre-training. find
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  24. Alejandro Ríos-Hoyo, Naing Lin Shan, Anran Li, Alexander T Pearson, Lajos Pusztai, and Frederick M Howard. 2024. Evaluation of large language models as a diagnostic aid for complex medical cases. Frontiers in Medicine, 11:1380148. find
  25. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416. find
  26. Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960. — [arXiv:2405.07960](https://arxiv.org/abs/2405.07960) find
  27. Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, Xiaofan Zhang, and Shaoting Zhang. 2024. Medical dialogue system: A survey of categories, methods, evaluation and challenges. Findings of the Association for Computational Linguistics ACL 2024, pages 2840–2861. find
  28. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8. find
  29. Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, et al. 2025. Towards conversational diagnostic artificial intelligence. Nature, pages 1–9. find
  30. Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. 2024. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology–a recent scoping review. Diagnostic pathology, 19(1):43. find
  31. Aditya B Vishwanath, Vijay Kumar Srinivasalu, and Narayana Subramaniam. 2024. Role of large language models in improving provider–patient experience and interaction efficiency: A scoping review. Artificial Intelligence in Health, page 4808. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-ask-patients-patience-enabling-llms/