Elamur Research⌕
☰
home / papers / 2023 / Large Language Models Perform Diagnostic Reasoning

Large Language Models Perform Diagnostic Reasoning

Paper · 2023 · Cheng-Kuang Wu, Wei-Lin Chen and Hsin-Hsi Chen

Summary

The paper asks whether general-purpose large language models can carry out the multi-step reasoning doctors use when diagnosing a patient through conversation. Earlier dialogue systems for automatic diagnosis mostly relied on reinforcement learning over structured symptom inputs and could handle only a handful of diseases. The authors build a few-shot dialogue system on InstructGPT and introduce Diagnostic-Reasoning CoT (DR-CoT). In this chain-of-thought variant, the model first restates the evidence collected so far, then produces a ranked differential diagnosis (DDx), and finally picks its next question to narrow that list. For evaluation, the same LLM also plays a scripted patient built from DDXPlus profiles, so doctor–patient conversations can be simulated automatically through self-chat. With only two exemplars, DR-CoT raises diagnostic accuracy by about 15% over standard prompting on in-domain cases and by about 18% on out-domain cases. This suggests that expert-style medical reasoning can be drawn out of models trained on general text using prompting alone. The authors stress that the results have not been validated in any clinical setting.

Key points

  • Model: InstructGPT text-davinci-003, accessed via the OpenAI API with temperature 0 and a 384-token limit. Dialogues are capped at T_max = 8 turns, after which an "[inform]" prefix forces the model to give a diagnosis.
  • Data: DDXPlus, which covers 49 pathologies and 223 evidence types, far more than older datasets with only 4–5 diagnoses. The test samples were grouped by initial evidence: cough for in-domain (200 cases), dyspnea for OD1 (100 cases) and runny nose for OD2 (100 cases).
  • Main result: DR-CoT outperforms standard prompting by about 15% in-domain and about 18% out-domain. It also reaches higher accuracy in fewer turns.
  • Patient-bot check: a physician reviewed 100 sampled question–answer pairs and judged 95 of the simulated patient replies faithful to the profile.
  • Human evaluation of question quality: the physician judged DR-CoT's questions more critical than standard prompting's in 18 vs. 2 dialogues (ID), 8 vs. 2 (OD1) and 9 vs. 1 (OD2).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan UniversityTaipei · Taiwan29 works · 2,116 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Cheng-Kuang Wu, Wei-Lin Chen and Hsin-Hsi Chen. 2023. Large Language Models Perform Diagnostic Reasoning. arXiv:2307.08922.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.08922 · arXiv 2307.08922

@misc{wu2023large,
  title = {{Large Language Models Perform Diagnostic Reasoning}},
  author = {Cheng-Kuang Wu, Wei-Lin Chen and Hsin-Hsi Chen},
  year = {2023},
  doi = {10.48550/arxiv.2307.08922},
  eprint = {2307.08922},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.08922}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. find
  2. Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. Approximating interactive human evaluation with self-play for open-domain dialog systems. Advances in Neural Information Processing Systems, 32, 2019. find
  3. Rebecca S Graves. Users’ guides to the medical literature: A manual for evidence-based clinical practice. Journal of the Medical Library Association, 90(4):483, 2002. find
  4. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
  5. Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087, 2019. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  6. Kangenbei Liao, Qianlong Liu, Zhongyu Wei, Baolin Peng, Qin Chen, Weijian Sun, and Xuanjing Huang. Task-oriented dialogue system for automatic disease diagnosis via hierarchical reinforcement learning. arXiv preprint arXiv:2004.14254, 2020. — [arXiv:2004.14254](https://arxiv.org/abs/2004.14254) find
  7. Wenge Liu, Yi Cheng, Hao Wang, Jianheng Tang, Yafei Liu, Ruihui Zhao, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. “my nose is running.” “are you also coughing?”: Building a medical diagnosis agent with interpretable inquiry logics. In Lud De Raedt (ed.), Proceedings of the Thirty-First International… — [doi:10.24963/ijcai.2022/592](https://doi.org/10.24963/ijcai.2022/592) find
  8. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384–1403, Abu Dhabi, United Arab Emirates, December 2022. Association for Com… find
  9. Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. Few-shot bot: Prompt-based learning for dialogue systems. arXiv preprint arXiv:2110.08118, 2021. — [arXiv:2110.08118](https://arxiv.org/abs/2110.08118) find
  10. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
  11. Jacqueline Rhoads, Julie C Penick, et al. Formulating a Differential Diagnosis for the Advanced Practice Provider. Springer Publishing Company, 2017. find
  12. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  13. Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. Ddxplus: A new dataset for automatic medical diagnosis. Proceedings of the Neural Information Processing Systems-Track on Datasets and Benchmarks, 2, 2022. find
  14. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
  15. Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong, and Xiangying Dai. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 201–207, M…
  16. Yuan Xia, Jingbo Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. Generative adversarial regularized mutual information policy gradient framework for automatic diagnosis. In Proceedings of the AAAI conference on artificial intelligence, pp. 1062–1069, 2020. find
  17. Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jianheng Tang, and Liang Lin. End-to-end knowledge-routed relational dialogue system for automatic diagnosis. In Proceedings of the AAAI conference on artificial intelligence, pp. 7346–7353, 2019. find
  18. Xinyan Zhao, Liangwei Chen, and Huanhuan Chen. A weighted heterogeneous graph-based dialog system. IEEE Transactions on Neural Networks and Learning Systems, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-large-language-models-perform-diagnostic/