Elamur Research⌕
☰
home / papers / 2023 / Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery

Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery

Paper · 2023 · Debadutta Dash, Rahul Thapa, Juan M. Banda et al.

Summary

Most tests of medical LLMs rely on exam-style benchmarks, which say little about real clinical use. This study asks whether GPT-3.5 and GPT-4 can safely and usefully answer questions physicians actually raised during patient care. The authors drew on questions sent to a Stanford informatics consultation service between 2017 and 2019. That service answered each question by analysing de-identified records of similar patients. After filtering, 66 questions remained. Each was sent to both models with only a short role-assigning prompt. Twelve physicians then judged every response on two criteria: whether it could harm a patient, and whether it matched the consult report's conclusion. A response counted as harmful or concordant only if more than six reviewers agreed. By majority vote, no response from either model was judged harmful. However, agreement with the consult reports was low, some responses cited invented references, and reviewers often reached no majority. The work offers an early template for evaluating LLMs at the point of care. It also suggests that grounding, prompt design and local adaptation are still required.

Key points

  • Question set: 154 consult questions (Feb 2017 – Aug 2019) were filtered down to 66. Excluded were questions lacking a clear PICOT form, questions tied specifically to Stanford patients, duplicates, and questions without a usable report.
  • Harm: By majority vote (>6 of 12 physicians), harmful responses were 0/66 for both GPT-3.5 and GPT-4. Inter-rater agreement on harm was very low (Fleiss kappa 0.0926).
  • Concordance with consult reports:
    • GPT-3.5: 8 agree, 20 disagree, 9 unable to assess, 29 with no majority.
    • GPT-4: 13 agree, 15 disagree, 3 unable to assess, 35 with no majority.
    • Agreement therefore rose by about 63% and disagreement fell by 25%.
  • Hallucinated references: GPT-3.5 produced 2 and GPT-4 produced 9. Using Bing Chat's 'More Precise' mode yielded answers without these fabricated citations.
  • Limitations: The question set is small and skewed toward internal medicine, dermatology and descriptive questions. The local EHR-based reports were treated as ground truth, which the models had no access to.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,189 citations🏢MicrosoftRedmond · United States404 works · 163,863 citations🎓Brown UniversityProvidence · United States85 works · 26,322 citations🎓University of OttawaOttawa · Canada14 works · 2,558 citations🏥Stanford MedicineStanford · United States6 works · 1,507 citations

Also on the byline: Stanford Health Care.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksMedicine · 288 worksHealth Informatics · 227 worksHealth Professions · 76 worksHealth Information Management · 57 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Debadutta Dash, Rahul Thapa, Juan M. Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H. Chen, Saurabh Gombar, Lance Downing, Rachel Pedreira, Ethan Goh, Angel Arnaout, Garret Kenn Morris, Honor Magon, Matthew P Lungren, Eric Horvitz and Nigam H. Shah. 2023. Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery. arXiv:2304.13714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.13714 · arXiv 2304.13714

@misc{dash2023evaluation,
  title = {{Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery}},
  author = {Debadutta Dash, Rahul Thapa, Juan M. Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H. Chen, Saurabh Gombar, Lance Downing, Rachel Pedreira, Ethan Goh, Angel Arnaout, Garret Kenn Morris, Honor Magon, Matthew P Lungren, Eric Horvitz and Nigam H. Shah},
  year = {2023},
  doi = {10.48550/arxiv.2304.13714},
  eprint = {2304.13714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.13714}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sevilla J, Heim L, Ho A, Besiroglu T, Hobbhahn M, Villalobos P. Compute Trends Across Three Eras of Machine Learning. arXiv [csLG]. Published online February 11, 2022. http://arxiv.org/abs/2202.05924 — [arXiv:2202.05924](https://arxiv.org/abs/2202.05924) ↗
  2. Bommasani R, Hudson DA, Adeli E, Altman R, Arora S, von Arx S, et al. On the Opportunities and Risks of Foundation Models. arXiv [csLG]. Published online August 16, 2021. http://arxiv.org/abs/2108.07258 ↗
  3. OpenAI. GPT-4 Technical Report. arXiv [csCL]. Published online March 15, 2023. http://arxiv.org/abs/2303.08774 ↗
  4. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv [csCV]. Published online April 13, 2022. http://arxiv.org/abs/2204.06125 ↗
  5. Zhou C, Li Q, Li C, Yu J, Liu Y, Wang G, et al. A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT. arXiv [csAI]. Published online February 18, 2023. http://arxiv.org/abs/2302.09419 ↗
  6. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-1901.
  7. Zhang T, Ladhak F, Durmus E, Liang P, McKeown K, Hashimoto TB. Benchmarking Large Language Models for News Summarization. arXiv [csCL]. Published online January 31, 2023. http://arxiv.org/abs/2301.13848 ↗
  8. Vilar D, Freitag M, Cherry C, Luo J, Ratnakar V, Foster G. Prompting PaLM for Translation: Assessing Strategies and Performance. arXiv [csCL]. Published online November 16, 2022. http://arxiv.org/abs/2211.09102 ↗
  9. Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, et al. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596(7873):583-589.
  10. Vemprala S, Bonatti R, Bucker A, Kapoor A. Chatgpt for robotics: Design principles and model abilities. Published online 2023. https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf ↗
  11. Castro Nascimento CM, Pimentel AS. Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. J Chem Inf Model. 2023;63(6):1649-1655. find
  12. Wornow M, Xu Y, Thapa R, Patel B, Steinberg E, Fleming S, et al. The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs. arXiv [csLG]. Published online March 22, 2023. http://arxiv.org/abs/2303.12961 — [arXiv:2303.12961](https://arxiv.org/abs/2303.12961) ↗
  13. Eysenbach G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers. JMIR Med Educ. 2023;9:e46885. find
  14. Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. NATO Adv Sci Inst Ser E Appl Sci. 2021;11(14):6421.
  15. Liévin V, Hother CE, Winther O. Can large language models reason about medical questions? arXiv [csCL]. Published online July 17, 2022. http://arxiv.org/abs/2207.08143 ↗
  16. Singhal K, Azizi S, Tu T, Sara Mahdavi S, Wei J, Chung HW, et al. Large Language Models Encode Clinical Knowledge. arXiv [csCL]. Published online December 26, 2022. http://arxiv.org/abs/2212.13138 ↗
  17. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198.
  18. Jeblick K, Schachtner B, Dexl J, Mittermeier A, Stüber AT, Topalis J, et al. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports. arXiv [csCL]. Published online December 30, 2022. http://arxiv.org/abs/2212.14882 — [arXiv:2212.14882](https://arxiv.org/abs/2212.14882) ↗
  19. Macdonald C, Adeloye D, Sheikh A, Rudan I. Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. J Glob Health. 2023;13:01003. find
  20. Adamson B, Waskom M, Blarre A, Kelly J, Krismer K, Nemeth S, et al. Approach to machine learning for extraction of real-world data variables from electronic health records. bioRxiv. Published online March 6, 2023. doi:10.1101/2023.03.02.23286522 — [doi:10.1101/2023.03.02.23286522](https://doi.org/10.1101/2023.03.02.23286522) find
  21. Raji D, Denton E, Bender EM, Hanna A, Paullada A. AI and the Everything in the Whole Wide World Benchmark. In: Vanschoren J, Yeung S, eds. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Vol 1. Curran; 2021. https://datasets-benchmarks-proceedings.neurips.c… ↗
  22. Narayanan A, Kapoor S. GPT-4 and professional benchmarks: the wrong answer to the wrong question. AI Snake Oil. Published March 20, 2023. https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks. Accessed March 29, 2023 ↗
  23. Wang A, Pruksachatkun Y, Nangia N, Singh A, Michael J, Hill F, et al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv [csCL]. Published online May 2, 2019. http://arxiv.org/abs/1905.00537 ↗
  24. Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv [csCL]. Published online June 9, 2022. http://arxiv.org/abs/2206.04615 ↗
  25. Gao L, Tow J, Biderman S, Lovering C, Phang J, Thite A, et al. EleutherAI/lm-Evaluation-Harness: v0.3.0.; 2022. doi:10.5281/zenodo.7413426 — [doi:10.5281/zenodo.7413426](https://doi.org/10.5281/zenodo.7413426) find
  26. Mrabet Y, Demner-Fushman D. HOLMS: Alternative Summary Evaluation with Large Language Models. In: Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics; 2020:5679-5688. find
  27. Liang P, Bommasani R, Lee T, Tsipras D, Soylu D, Yasunaga M, et al. Holistic Evaluation of Language Models. arXiv [csCL]. Published online November 16, 2022. http://arxiv.org/abs/2211.09110 ↗
  28. Lee P, Bubeck S, Petro J, Eng. M. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. find
  29. Verghese A, Shah NH, Harrington RA. What This Computer Needs Is a Physician: Humanism and Artificial Intelligence. JAMA. 2018;319(1):19-20. find
  30. Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on Medical Challenge Problems. arXiv [csCL]. Published online March 20, 2023. http://arxiv.org/abs/2303.13375 ↗
  31. Callahan Alison, Gombar Saurabh, Cahan Eli M., Jung Kenneth, Steinberg Ethan, Polony Vladimir, et al. Using Aggregate Patient Data at the Bedside via an On-Demand Consultation Service. NEJM Catalyst. 2(10). doi:10.1056/CAT.21.0224 — [doi:10.1056/cat.21.0224](https://doi.org/10.1056/cat.21.0224) find
  32. Callahan Alison, Gombar Saurabh, Cahan Eli M., Jung Kenneth, Steinberg Ethan, Polony Vladimir, et al. Using Aggregate Patient Data at the Bedside via an On-Demand Consultation Service. NEJM Catalyst. 2(10). doi:10.1056/CAT.21.0224 — [doi:10.1056/cat.21.0224](https://doi.org/10.1056/cat.21.0224) find
  33. Riva JJ, Malik KMP, Burnie SJ, Endicott AR, Busse JW. What is your research question? An introduction to the PICOT format for clinicians. J Can Chiropr Assoc. 2012;56(3):167-171. find
  34. How Well Do Large Language Models Support Clinician Information Needs? Stanford HAI. https://hai.stanford.edu/news/how-well-do-large-language-models-support-clinician-informat ion-needs. Accessed April 18, 2023 ↗
  35. Qiu J, Li L, Sun J, Peng J, Shi P, Zhang R, et al. Large AI Models in Health Informatics: Applications, Challenges, and the Future. Published online March 21, 2023. http://arxiv.org/abs/2303.11568. Accessed April 24, 2023 — [arXiv:2303.11568](https://arxiv.org/abs/2303.11568) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dash-2023-evaluation-gpt-3-5-gpt/