📄 A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

Paper · 2023 · Hongjian Zhou, Fenglin Liu, Boyang Gu et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🎓University of OxfordOxford · United Kingdom52 works · 154,829 citations🎓Harvard UniversityCambridge · United States73 works · 37,309 citations🏢TencentShenzhen · China66 works · 17,945 citations🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓University College LondonLondon · United Kingdom59 works · 16,463 citations🏢AmazonSeattle · United States70 works · 39,977 citations🎓Yale UniversityNew Haven · United States54 works · 11,964 citations🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations🎓University of RochesterRochester · United States25 works · 6,799 citations🎓Imperial College LondonLondon · United Kingdom24 works · 8,272 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China7 works · 1,026 citations🎓University of GeorgiaAthens · United States9 works · 26,161 citations🎓Western UniversityLondon · Canada4 works · 204 citations

Cite

Cite as Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Chenyu You, Xian Wu, Yefeng Zheng, Lei Clifton, Zheng Li, Jiebo Luo and David A. Clifton. 2023. A Survey of Large Language Models in Medicine: Progress, Application, and Challenge. arXiv:2311.05112.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.05112 · arXiv 2311.05112

@misc{zhou2023survey,
  title = {{A Survey of Large Language Models in Medicine: Progress, Application, and Challenge}},
  author = {Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Chenyu You, Xian Wu, Yefeng Zheng, Lei Clifton, Zheng Li, Jiebo Luo and David A. Clifton},
  year = {2023},
  doi = {10.48550/arxiv.2311.05112},
  eprint = {2311.05112},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.05112}
}

References

271 references, 81 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zhao, W. X. et al. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). our card
2. Yang, J. et al. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712 (2023). our card
3. Chowdhery, A. et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). our card
4. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). our card
5. Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). find
6. Brown, T. et al. Language models are few-shot learners. Adv. neural information processing systems 33, 1877–1901 (2020). our card
7. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). our card
8. Du, Z. et al. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 320–335 (2022). our card
9. Zeng, A. et al. Glm-130b: An open bilingual pre-trained model. In International Conference on Learning Representations (2022). our card
10. Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). our card
11. Singhal, K. et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617 (2023). our card
12. Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 (2023). find
13. Wu, C., Zhang, X., Zhang, Y., Wang, Y. & Xie, W. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023). our card
14. Jin, D. et al. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021). our card
15. Li, Y. et al. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. arXiv preprint arXiv:2303.14070 (2023). our card
16. Han, T. et al. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247 (2023). our card
17. Wang, H. et al. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975 (2023). our card
18. Toma, A. et al. Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. arXiv preprint arXiv:2305.12031 (2023). find
19. Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. medicine 29, 1930–1940 (2023). find
20. Patel, S. B. & Lam, K. Chatgpt: the future of discharge summaries? The Lancet Digit. Heal. 5, e107–e108 (2023). find
21. Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. & Daneshjou, R. Large language models in medicine: The potentials and pitfalls. Annals Intern. Medicine (2024). find
22. Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Medicine 3, 141 (2023). find
23. Yang, X. et al. A large language model for electronic health records. NPJ Digit. Medicine 5, 194 (2022). find
24. Abd-Alrazaq, A. et al. Large language models in medical education: Opportunities, challenges, and future directions. JMIR Med. Educ. 9, e48291 (2023). find
25. Bengio, Y., Ducharme, R. & Vincent, P. A neural probabilistic language model. Adv. neural information processing systems 13 (2000). find
26. Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J. & Khudanpur, S. Recurrent neural network based language model. In Interspeech, vol. 2, 1045–1048 (2010). find
27. Sundermeyer, M., Ney, H. & Schlüter, R. From feedforward to recurrent lstm neural networks for language modeling. IEEE/ACM Transactions on Audio, Speech, Lang. Process. 23, 517–529 (2015). find
28. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997). find
29. Vaswani, A. et al. Attention is all you need. Adv. neural information processing systems 30 (2017). our card
30. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). our card
31. Kaplan, J. et al. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). our card
32. Hoffmann, J. et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022). our card
33. He, P., Liu, X., Gao, J. & Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2021). our card
34. Google. Bard: A generative artificial intelligence chatbot. https://gemini.google.com (2023). ↗
35. Taori, R. et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca (2023). ↗
36. Yang, A. et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 (2023). our card
37. Chung, H. W. et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). our card
38. Joseph, S. et al. Multilingual simplification of medical texts. arXiv preprint arXiv:2305.12532 (2023). find
39. Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). find
40. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019). find
41. Chiang, W.-L. et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality (2023). find
42. Jiang, A. Q. et al. Mistral 7b. arXiv preprint arXiv:2310.06825 (2023). find
43. Meta llama 3. https://github.com/meta-llama/llama3 (2024). ↗
44. Bai, J. et al. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023). our card
45. Ouyang, L. et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 35, 27730–27744 (2022). our card
46. Claude. https://www.anthropic.com/claude (2024). ↗
47. Lewis, M. et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019). our card
48. Tay, Y. et al. Ul2: Unifying language learning paradigms. In International Conference on Learning Representations (2022). our card
49. Lee, J. et al. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020). our card
50. National Institutes of Health. PubMed Corpora (https://pubmed.ncbi.nlm.nih.gov/download/). In National Library of Medicine (2022). 51. https://www.ncbi.nlm.nih.gov/pmc/. ↗
51. Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Comput. for Healthc. (HEALTH) 3, 1–23 (2021). our card
52. Beltagy, I., Lo, K. & Cohan, A. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019). our card
53. Ammar, W. et al. Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262 (2018). our card
54. Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619, 357–362 (2023). find
55. Alsentzer, E. et al. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019). our card
56. Johnson, A. E. et al. Mimic-iii, a freely accessible critical care database. Sci. data 3, 1–9 (2016). our card
57. Alrowili, S. & Shanker, V. Large biomedical question answering models with albert and electra. In CLEF (Working Notes), 213–220 (2021). find
58. Gururangan, S. et al. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of Association for Computational Linguistics (ACL) (2020). find
59. Lo, K., Wang, L. L., Neumann, M., Kinney, R. & Weld, D. S. S2orc: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782 (2019). our card
60. Yasunaga, M., Leskovec, J. & Liang, P. Linkbert: Pretraining language models with document links. In Proceedings of Association for Computational Linguistics (ACL) (2022). our card
61. Peng, Y., Yan, S. & Lu, Z. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, 58–65 (2019). our card
62. Mutinda, F. W. et al. Detecting redundancy in electronic medical records using clinical bert. In Proceedings of the Annual Conference of the Association for Natural Language Processing, 16–19 (2020). find
63. Mahajan, D. et al. Identification of semantically similar sentences in clinical notes: Iterative intermediate training using multi-task learning. JMIR medical informatics 8, e22508 (2020). find
64. Phan, L. N. et al. Scifive: a text-to-text transformer model for biomedical literature. arXiv preprint arXiv:2106.03598 (2021). — [arXiv:2106.03598](https://arxiv.org/abs/2106.03598) find
65. Lu, Q., Dou, D. & Nguyen, T. Clinicalt5: A generative language model for clinical text. In Findings of the Association for Computational Linguistics: EMNLP 2022, 5436–5443 (2022). find
66. Jin, Q. et al. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. arXiv preprint arXiv:2307.00589 (2023). — [arXiv:2307.00589](https://arxiv.org/abs/2307.00589) find
67. Yasunaga, M. et al. Deep bidirectional language-knowledge graph pretraining. Adv. Neural Inf. Process. Syst. 35, 37309–37323 (2022). find
68. Zhu, Y. et al. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 19–27 (2015). our card
69. Luo, R. et al. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings Bioinforma. 23, bbac409 (2022). find
70. Venigalla, A., Frankle, J. & Carbin, M. Biomedlm: a domain-specific large language model for biomedical text. MosaicML. Accessed: Dec 23, 2 (2022). find
71. Gao, L. et al. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). find
72. Gao, W. et al. Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue. arXiv preprint arXiv:2306.12174 (2023). — [arXiv:2306.12174](https://arxiv.org/abs/2306.12174) find
73. Chen, S. et al. Meddialog: a large-scale medical dialogue dataset. arXiv preprint arXiv:2004.03329 3 (2020). — [arXiv:2004.03329](https://arxiv.org/abs/2004.03329) find
74. Peng, C. et al. A study of generative large language model for medical research and healthcare. arXiv preprint arXiv:2305.13523 (2023). find
75. Xiong, H. et al. Doctorglm: Fine-tuning your chinese doctor is not a herculean task. arXiv preprint arXiv:2304.01097 (2023). find
76. Toyhom. Chinese medical dialogue data. https://github.com/Toyhom/Chinese-medical-dialogue-data (2023). GitHub repository. ↗
77. Chen, Y. et al. Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt. arXiv preprint arXiv:2310.15896 (2023). — [arXiv:2310.15896](https://arxiv.org/abs/2310.15896) find
78. Wang, G., Yang, G., Du, Z., Fan, L. & Li, X. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation. arXiv preprint arXiv:2306.09968 (2023). find
79. Ye, Q. et al. Qilin-med: Multi-stage knowledge injection advanced medical large language model. arXiv preprint arXiv:2310.09089 (2023). — [arXiv:2310.09089](https://arxiv.org/abs/2310.09089) find
80. Healthcaremagic. https://www.healthcaremagic.com. 82. https://www.icliniq.com/. ↗
81. Byambasuren, O. et al. Preliminary study on the construction of chinese medical knowledge graph. J. Chin. Inf. Process. 33, 1–9 (2019). find
82. Zhang, H. et al. Huatuogpt, towards taming language model to be a doctor. arXiv preprint arXiv:2305.15075 (2023). our card
83. Xu, C., Guo, D., Duan, N. & McAuley, J. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196 (2023). our card
84. Abacha, A. B. & Demner-Fushman, D. A question-entailment approach to question answering. BMC Bioinforma. 20 (2019). find
85. Luo, Y. et al. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442 (2023). — [arXiv:2308.09442](https://arxiv.org/abs/2308.09442) find
86. Zhang, X. et al. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558 (2023). — [arXiv:2310.14558](https://arxiv.org/abs/2310.14558) find
87. Yang, S. et al. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. arXiv preprint arXiv:2308.03549 (2023). — [arXiv:2308.03549](https://arxiv.org/abs/2308.03549) find
88. Shoham, O. B. & Rappoport, N. Cpllm: Clinical prediction with large language models. arXiv preprint arXiv:2309.11295 (2023). — [arXiv:2309.11295](https://arxiv.org/abs/2309.11295) find
89. Pollard, T. J. et al. The eicu collaborative research database, a freely available multi-center database for critical care research. Sci. data 5, 1–13 (2018). find
90. Johnson, A. et al. Mimic-iv. https://physionet.org/content/mimiciv/1.0/ (2020). ↗
91. Ferber, D. et al. Gpt-4 for information retrieval and comparison of medical oncology guidelines. NEJM AI 1, AIcs2300235 (2024). find
92. Chen, Z. et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023). our card
93. Bosselut, A. et al. Meditron: Open medical foundation models adapted for clinical practice. Preprint (2024). find
94. Ankit Pal, M. S. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B (2024). ↗
95. Medllama3-v20. https://huggingface.co/ProbeMedicalYonseiMAILab/medllama3-v20 (2024). ↗
96. Sharegpt: Share your wildest chatgpt conversations with one click. https://sharegpt.com (2023). ↗
97. Moor, M. et al. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), 353–367 (2023). find
98. Lau, J. J., Gayen, S., Ben Abacha, A. & Demner-Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Sci. data 5, 1–10 (2018). find
99. He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020). — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) find
100. Li, C. et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Adv. Neural Inf. Process. Syst. 36 (2024). our card
101. Liu, B. et al. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650–1654 (IEEE, 2021). find
102. Hyland, S. L. et al. Maira-1: A specialised large multimodal model for radiology report generation. arXiv preprint arXiv:2311.13668 (2023). — [arXiv:2311.13668](https://arxiv.org/abs/2311.13668) find
103. Johnson, A. E. et al. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Sci. data 6, 317 (2019). find
104. Wu, C., Zhang, X., Zhang, Y., Wang, Y. & Xie, W. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463 (2023). find
105. Yang, L. et al. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03162 (2024). — [arXiv:2405.03162](https://arxiv.org/abs/2405.03162) find
106. Saab, K. et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024). find
107. Tanno, R. et al. Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation. arXiv preprint arXiv:2311.18260 (2023). — [arXiv:2311.18260](https://arxiv.org/abs/2311.18260) find
108. Liévin, V., Hother, C. E. & Winther, O. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143 (2022). find
109. Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 35, 24824–24837 (2022). our card
110. Liu, Z. et al. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032 (2023). — [arXiv:2303.11032](https://arxiv.org/abs/2303.11032) find
111. Wang, S., Zhao, Z., Ouyang, X., Wang, Q. & Shen, D. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257 (2023). — [arXiv:2302.07257](https://arxiv.org/abs/2302.07257) find
112. Gao, Y. et al. Leveraging a medical knowledge graph into large language models for diagnosis prediction. arXiv e-prints arXiv–2308 (2023). find
113. Bodenreider, O. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research 32, D267–D270 (2004). find
114. Shi, W. et al. Retrieval-augmented large language models for adolescent idiopathic scoliosis patients in shared decision-making. In Proceedings of the 14th ACM International Conference on BCB, 1–10 (2023). find
115. SRS. https://www.srs.org. Accessed: 2024-05-14. ↗
116. UpToDate. http://uptodate.com. Accessed: 2024-05-14. ↗
117. Dynamed. https://www.dynamed.com. Accessed: 2024-05-14. ↗
118. Kim, J. & Min, M. From rag to qa-rag: Integrating generative ai for pharmaceutical regulatory compliance process. arXiv preprint arXiv:2402.01717 (2024). find
119. Zakka, C. et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1, AIoa2300068 (2024). find
120. Wu, J. et al. Clinical text datasets for medical artificial intelligence and large language models—a systematic review. NEJM AI 1, AIra2400012 (2024). find
121. He, K. et al. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694 (2023). our card
122. Zhang, S. et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792 (2023). our card
123. Wang, H., Liu, C., Zhao, S., Qin, B. & Liu, T. Chatglm-med. https://github.com/SCIR-HI/Med-ChatGLM (2023). ↗
124. Hu, E. J. et al. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021). our card
125. Li, X. L. & Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021). find
126. Liu, X. et al. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 61–68 (2022). our card
127. Liu, X. et al. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021). our card
128. Houlsby, N. et al. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, 2790–2799 (2019). find
129. Dong, Q. et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022). our card
130. Lester, B., Al-Rfou, R. & Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059 (2021). find
131. Gao, Y. et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023). our card
132. Xiong, G., Jin, Q., Lu, Z. & Zhang, A. Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178 (2024). our card
133. Li, X. & Li, J. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871 (2023). our card
134. Wang, G. et al. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023). find
135. Chen, J. et al. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2309.07597 (2023). our card
136. Shao, Z. et al. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305.15294 (2023). our card
137. Trivedi, H., Balasubramanian, N., Khot, T. & Sabharwal, A. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509 (2022). our card
138. Asai, A., Wu, Z., Wang, Y., Sil, A. & Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 (2023). our card
139. Donnelly, K. et al. Snomed-ct: The advanced terminology and coding system for ehealth. Stud. health technology informatics 121, 279 (2006). find
140. Organization, W. H. et al. International classification of diseases:[9th] ninth revision, basic tabulation list with alphabetic index (World Health Organization, 1978). find
141. Tang, L. et al. Evaluating large language models on medical evidence summarization. npj Digit. Medicine 6, 158 (2023). find
142. Van Veen, D. et al. Clinical text summarization: Adapting large language models can outperform human experts. arXiv preprint arXiv:2309.07430 (2023). our card
143. Ondov, B., Attal, K. & Demner-Fushman, D. A survey of automated methods for biomedical text simplification. J. Am. Med. Informatics Assoc. 29, 1976–1988 (2022). find
144. Liu, F. et al. Retrieve, reason, and refine: Generating accurate and faithful patient instructions. Adv. Neural Inf. Process. Syst. 35, 18864–18877 (2022). find
145. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019). our card
146. Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, 248–260 (2022). our card
147. Doğan, R. I., Leaman, R. & Lu, Z. Ncbi disease corpus: a resource for disease name recognition and concept normalization. J. biomedical informatics 47, 1–10 (2014). find
148. Dong, H. et al. Automated clinical coding: what, why, and where we are? NPJ digital medicine 5, 159 (2022). find
149. D’Onofrio, G. et al. Emotion recognizing by a robotic solution initiative. Sensors 22, 2861 (2022). find
150. Biri, S. K. et al. Assessing the utilization of large language models in medical education: Insights from undergraduate medical students. Cureus 15 (2023). find
151. Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S. & Torous, J. B. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. The Can. J. Psychiatry 64, 456–464 (2019). find
152. McDuff, D. et al. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164 (2023). — [arXiv:2312.00164](https://arxiv.org/abs/2312.00164) find
153. Kraljevic, Z. et al. Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. The Lancet Digit. Heal. 6, e281–e290 (2024). find
154. Jin, Q. et al. Matching patients to clinical trials with large language models. ArXiv (2023). find
155. Koopman, B. & Zuccon, G. A test collection for matching patients to clinical trials. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 669–672 (2016). find
156. Roberts, K., Demner-Fushman, D., Voorhees, E. M., Bedrick, S. & Hersh, W. R. Overview of the trec 2022 clinical trials track. In TREC (2022). find
157. Huang, C.-W., Tsai, S.-C. & Chen, Y.-N. Plm-icd: Automatic icd coding with pretrained language models. arXiv e-prints arXiv–2207 (2022). find
158. Saeed, M., Lieu, C., Raber, G. & Mark, R. G. Mimic ii: a massive temporal icu patient database to support research in intelligent patient monitoring. In Computers in cardiology, 641–644 (IEEE, 2002). find
159. Wang, H., Gao, C., Dantona, C., Hull, B. & Sun, J. Drg-llama: tuning llama model to predict diagnosis-related group for hospitalized patients. npj Digit. Medicine 7, 16 (2024). find
160. Liu, J., Yang, S., Peng, T., Hu, X. & Zhu, Q. Chaticd: Prompt learning for few-shot icd coding through chatgpt. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 4360–4367 (2023). find
161. Yang, Z., Batra, S. S., Stremmel, J. & Halperin, E. Surpassing gpt-4 medical coding with a two-stage approach. arXiv preprint arXiv:2311.13735 (2023). — [arXiv:2311.13735](https://arxiv.org/abs/2311.13735) find
162. Ma, C. et al. An iterative optimizing framework for radiology report summarization with chatgpt. IEEE Transactions on Artif. Intell. (2024). find
163. Open-i. https://openi.nlm.nih.gov/. Accessed: 2024-05-14. ↗
164. Van Veen, D. et al. Radadapt: Radiology report summarization via lightweight domain adaptation of large language models. arXiv preprint arXiv:2305.01146 (2023). find
165. Moghani, M. et al. Sufia: Language-guided augmented dexterity for robotic surgical assistants. arXiv preprint arXiv:2405.05226 (2024). — [arXiv:2405.05226](https://arxiv.org/abs/2405.05226) find
166. Yu, Q. et al. Orbit-surgical: An open-simulation framework for learning surgical augmented dexterity. arXiv preprint arXiv:2404.16027 (2024). find
167. Xu, H. et al. Enhancing surgical robots with embodied intelligence for autonomous ultrasound scanning. arXiv preprint arXiv:2405.00461 (2024). — [arXiv:2405.00461](https://arxiv.org/abs/2405.00461) find
168. Killeen, B. D., Chaudhary, S., Osgood, G. & Unberath, M. Take a shot! natural language control of intelligent robotic x-ray systems in surgery. Int. J. Comput. Assist. Radiol. Surg. 1–9 (2024). find
169. García-Ferrero, I. et al. Medical mt5: an open-source multilingual text-to-text llm for the medical domain. arXiv preprint arXiv:2404.07613 (2024). — [arXiv:2404.07613](https://arxiv.org/abs/2404.07613) find
170. Tiedemann, J. Parallel data, tools and interfaces in opus. In Lrec, vol. 2012, 2214–2218 (2012). find
171. National Library of Medicine. Clinical trials. https://clinicaltrials.gov/ (2022). Accessed: 2024-05-14. ↗
172. Wang, X. et al. Apollo: Lightweight multilingual medical llms towards democratizing medical ai to 6b people. arXiv preprint arXiv:2403.03640 (2024). — [arXiv:2403.03640](https://arxiv.org/abs/2403.03640) find
173. Pieri, S. et al. Bimedix: Bilingual medical mixture of experts llm. arXiv preprint arXiv:2402.13253 (2024). find
174. Tang, C., Wang, S., Goldsack, T. & Lin, C. Improving biomedical abstractive summarisation with knowledge aggregation from citation papers. arXiv preprint arXiv:2310.15684 (2023). find
175. Guo, Y., Qiu, W., Leroy, G., Wang, S. & Cohen, T. Retrieval augmentation of large language models for lay language generation. J. Biomed. Informatics 149, 104580 (2024). find
176. Peacock, J., Austin, A., Shapiro, M., Battista, A. & Samuel, A. Accelerating medical education with chatgpt: an implementation guide. MedEdPublish 13 (2023). find
177. Qiu, H., Li, A., Ma, L. & Lan, Z. Psychat: A client-centric dialogue system for mental health support. arXiv preprint arXiv:2312.04262 (2023). — [arXiv:2312.04262](https://arxiv.org/abs/2312.04262) find
178. Liu, J. M. et al. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:2309.15461 (2023). — [arXiv:2309.15461](https://arxiv.org/abs/2309.15461) find
179. Xu, X. et al. Mental-llm: Leveraging large language models for mental health prediction via online text data. Proc. ACM on Interactive, Mobile, Wearable Ubiquitous Technol. 8, 1–32 (2024). find
180. Turcan, E. & McKeown, K. Dreaddit: A reddit dataset for stress analysis in social media. arXiv preprint arXiv:1911.00133 (2019). find
181. Naseem, U., Dunn, A. G., Kim, J. & Khushi, M. Early identification of depression severity levels on reddit using ordinal classification. In Proceedings of the ACM Web Conference 2022, 2563–2572 (2022). find
182. Haque, A., Reddi, V. & Giallanza, T. Deep learning for suicide and depression identification with unsupervised label correction. In International Conference on Artificial Neural Networks, 436–447 (2021). find
183. Gaur, M. et al. Knowledge-aware assessment of severity of suicide risk for early intervention. In The world wide web conference, 514–525 (2019). find
184. Sampath, K. & Durairaj, T. Data set creation and empirical analysis for detecting signs of depression from social media postings. In International Conference on Computational Intelligence in Data Science, 136–151 (2022). find
185. Jamil, Z. Monitoring tweets for depression to detect at-risk users. Ph.D. thesis, Université d’Ottawa/University of Ottawa (2017). find
186. Mauriello, M. L. et al. Sad: A stress annotated dataset for recognizing everyday stressors in sms-like conversational systems. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, 1–7 (2021). find
187. Tu, T. et al. Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654 (2024). — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) find
188. Ren, Z., Zhan, Y., Yu, B., Ding, L. & Tao, D. Healthcare copilot: Eliciting the power of general llms for medical consultation. arXiv preprint arXiv:2402.13408 (2024). — [arXiv:2402.13408](https://arxiv.org/abs/2402.13408) find
189. Sun, Z., Luo, C. & Huang, Z. Conversational disease diagnosis via external planner-controlled large language models. arXiv preprint arXiv:2404.04292 (2024). — [arXiv:2404.04292](https://arxiv.org/abs/2404.04292) find
190. Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. Ai in health and medicine. Nat. medicine 28, 31–38 (2022). find
191. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. The J. Mach. Learn. Res. 21, 5485–5551 (2020). our card
192. Liu, F., Wu, X., Ge, S., Fan, W. & Zou, Y. Exploring and distilling posterior and prior knowledge for radiology report generation. In IEEE Conference on Computer Vision and Pattern Recognition (2021). find
193. Food, Administration, D. et al. National drug code directory (Consumer Protection and Environmental Health Service, Public Health Service . . . , 1976). find
194. Liu, S., Ma, W., Moore, R., Ganesan, V. & Nelson, S. Rxnorm: prescription for electronic drug information exchange. IT professional 7, 17–23 (2005). find
195. Elkin, P. L. & Brown, S. H. Current procedural terminology. In Terminology, Ontology and their Implementations, 367–370 (Springer, 2023). find
196. Liu, X. et al. Reporting guidelines for clinical trials evaluating artificial intelligence interventions are needed. Nat. Medicine 25, 1467–1469 (2019). find
197. Papineni, K., Roukos, S., Ward, T. & Zhu, W. BLEU: a Method for automatic evaluation of machine translation. In Proceedings of Association for Computational Linguistics (ACL) (2002). our card
198. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Proceedings of Association for Computational Linguistics (ACL) (2004). our card
199. Banerjee, S. & Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and summarization, 65–72 (2005). our card
200. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). find
201. Smit, A. et al. Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint arXiv:2004.09167 (2020). our card
202. Jain, S. et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463 (2021). — [arXiv:2106.14463](https://arxiv.org/abs/2106.14463) find
203. Yu, F. et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns 4 (2023). find
204. Xie, Q. et al. Faithful ai in medicine: A systematic review with large language models and beyond. medRxiv (2023). find
205. Dupont, P. E. et al. A decade retrospective of medical robotics research from 2010 to 2020. Sci. robotics 6, eabi8017 (2021). find
206. Chen, K. et al. Llm-assisted multi-teacher continual learning for visual question answering in robotic surgery. arXiv preprint arXiv:2402.16664 (2024). — [arXiv:2402.16664](https://arxiv.org/abs/2402.16664) find
207. Wang, J. et al. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv preprint arXiv:2401.04334 (2024). — [arXiv:2401.04334](https://arxiv.org/abs/2401.04334) find
208. Weerarathna, I. N., Raymond, D. & Luharia, A. Human-robot collaboration for healthcare: A narrative review. Cureus 15 (2023). find
209. Moglia, A., Georgiou, K., Georgiou, E., Satava, R. M. & Cuschieri, A. A systematic review on artificial intelligence in robot-assisted surgery. Int. J. Surg. 95, 106151 (2021). find
210. Xia, Y., Wang, S. & Kan, Z. A nested u-structure for instrument segmentation in robotic surgery. In International Conference on Advanced Robotics and Mechatronics (ICARM), 994–999 (2023). find
211. Chen, Y., Arunasalam, A. & Celik, Z. B. Can large language models provide security & privacy advice? measuring the ability of llms to refute misconceptions. In Proceedings of the 39th Annual Computer Security Applications Conference, 366–378 (2023). find
212. Karabacak, M. et al. The advent of generative language models in medical education. JMIR Med. Educ. 9, e48163 (2023). find
213. Qiu, J. et al. Large ai models in health informatics: Applications, challenges, and the future. IEEE J. Biomed. Heal. Informatics (2023). find
214. Ahn, S. The impending impacts of large language models on medical education. Korean J. Med. Educ. 35, 103 (2023). find
215. OpenAI. Chatgpt [large language model]. https://chat.openai.com (2023). ↗
216. Rawte, V., Sheth, A. & Das, A. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 (2023). our card
217. Stock, A., Schlögl, S. & Groth, A. Tell me, what are you most afraid of? exploring the effects of agent representation on information disclosure in human-chatbot interaction. arXiv e-prints arXiv–2307 (2023). find
218. De Choudhury, M., Pendse, S. R. & Kumar, N. Benefits and harms of large language models in digital mental health. arXiv preprint arXiv:2311.14693 (2023). — [arXiv:2311.14693](https://arxiv.org/abs/2311.14693) find
219. Hua, Y. et al. Large language models in mental health care: a scoping review (2024). 2401.02984. find
220. Robinson, N., Connolly, J., Suddrey, G. & Kavanagh, D. J. A brief wellbeing training session delivered by a humanoid social robot: A pilot randomized controlled trial. arXiv e-prints arXiv–2308 (2023). find
221. Lai, T. et al. Psy-llm: Scaling up global mental health psychological services with ai-based large language models. arXiv preprint arXiv:2307.11991 (2023). — [arXiv:2307.11991](https://arxiv.org/abs/2307.11991) find
222. Ma, Z., Mei, Y. & Su, Z. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. In AMIA Annual Symposium Proceedings, vol. 2023, 1105 (2023). find
223. Chung, N. C., Dyer, G. & Brocki, L. Challenges of large language models for mental health counseling. arXiv preprint arXiv:2311.13857 (2023). — [arXiv:2311.13857](https://arxiv.org/abs/2311.13857) find
224. Wang, J., Yang, Z., Yao, Z. & Yu, H. Jmlr: Joint medical llm and retrieval training for enhancing reasoning and professional question answering capability. arXiv preprint arXiv:2402.17887 (2024). find
225. Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Medicine 1–10 (2024). find
226. Stokel-Walker, C. Chatgpt listed as author on research papers: many scientists disapprove. Nature 613, 620–621 (2023). find
227. Shen, X., Chen, Z., Backes, M., Shen, Y. & Zhang, Y. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825 (2023). our card
228. Umapathi, L. K., Pal, A. & Sankarasubbu, M. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 (2023). our card
229. Roit, P. et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186 (2023). find
230. Chern, I.-C. et al. Improving factuality of abstractive summarization via contrastive reward learning. arXiv preprint arXiv:2307.04507 (2023). find
231. Manakul, P., Liusie, A. & Gales, M. J. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023). find
232. Shuster, K., Poff, S., Chen, M., Kiela, D. & Weston, J. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567 (2021). our card
233. Dhuliawala, S. et al. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023). our card
234. Lin, S., Hilton, J. & Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 (2021). our card
235. Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y. & Wen, J.-R. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv e-prints arXiv–2305 (2023). our card
236. Liu, F. et al. Auto-encoding knowledge graph for unsupervised medical report generation. In Advances in Neural Information Processing Systems (2021). find
237. Shumailov, I. et al. Model dementia: Generated data makes models forget. arXiv preprint arXiv:2305.17493 (2023). our card
238. Hoelscher-Obermaier, J., Persson, J., Kran, E., Konstas, I. & Barez, F. Detecting edit failures in large language models: An improved specificity benchmark. arXiv preprint arXiv:2305.17553 (2023). find
239. Liu, F. et al. A medical multimodal large language model for future pandemics. npj Digit. Medicine 6, 226 (2023). find
240. Yao, Y. et al. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172 (2023). find
241. Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv. Neural Inf. Process. Syst. 33, 9459–9474 (2020). find
242. Hendrycks, D. et al. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275 (2020). our card
243. Glaese, A. et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 (2022). our card
244. Nakano, R. et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021). our card
245. Liu, H., Sferrazza, C. & Abbeel, P. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02676 3 (2023). find
246. Sallam, M. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, 887 (MDPI, 2023). find
247. Tian, S. et al. Opportunities and challenges for chatgpt and large language models in biomedicine and health. Briefings Bioinforma. 25, bbad493 (2024). find
248. Li, H., Guo, D., Fan, W., Xu, M. & Song, Y. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197 (2023). our card
249. Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483 (2023). our card
250. Meskó, B. & Topol, E. J. The imperative for regulatory oversight of large language models (or generative ai) in healthcare. NPJ digital medicine 6, 120 (2023). find
251. Derraz, B. et al. New regulatory thinking is needed for ai-based personalised drug and cell therapies in precision oncology. NPJ Precis. Oncol. 8, 23 (2024). find
252. Hacker, P., Engel, A. & Mauer, M. Regulating chatgpt and other large generative ai models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1112–1123 (2023). find
253. Mökander, J., Schuett, J., Kirk, H. R. & Floridi, L. Auditing large language models: a three-layered approach. AI Ethics 1–31 (2023). find
254. Chen, Q. et al. An extensive benchmark study on biomedical text generation and mining with chatgpt. Bioinformatics 39, btad557 (2023). find
255. Chen, Q. et al. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv preprint arXiv:2305.16326 (2023). find
256. Yin, S. et al. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023). our card
257. Tu, T. et al. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334 (2023). find
258. Li, C. et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 (2023). our card
259. Shu, C., Liu, F. & Shareghi, C. Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities. https://github.com/cambridgeltl/visual-med-alpaca (2023). ↗
260. Moor, M. et al. Med-flamingo: a multimodal medical few-shot learner. arXiv preprint arXiv:2307.15189 (2023). find
261. Liu, J. et al. Qilin-med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint arXiv:2310.17956 (2023). — [arXiv:2310.17956](https://arxiv.org/abs/2310.17956) find
262. Huang, H. et al. Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model. Int. J. Oral Sci. 15, 29 (2023). find
263. Li, J., Liu, C., Cheng, S., Arcucci, R. & Hong, S. Frozen language model helps ecg zero-shot learning. arXiv preprint arXiv:2303.12311 (2023). — [arXiv:2303.12311](https://arxiv.org/abs/2303.12311) find
264. Englhardt, Z. et al. Exploring and characterizing large language models for embedded system development and debugging. arXiv preprint arXiv:2307.03817 (2023). — [arXiv:2307.03817](https://arxiv.org/abs/2307.03817) find
265. Xi, Z. et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023). our card
266. Wang, L. et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 (2023). our card
267. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D. & Ghanem, B. Camel: Communicative agents for "mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760 (2023). our card
268. Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.10537 (2023). our card
269. Organization, W. H. Physical activity (2022). Accessed: Aug. 18, 2023. find
270. Connor, M. & O’Neill, M. Large language models in sport science & medicine: Opportunities, risks and considerations. arXiv preprint arXiv:2305.03851 (2023). — [arXiv:2305.03851](https://arxiv.org/abs/2305.03851) find
271. Mello, M. M. & Guha, N. Chatgpt and physicians’ malpractice risk. In JAMA Health Forum, e231938–e231938 (2023). \#\# Acknowledgements This work was supported in part by the Pandemic Sciences Institute at the University of Oxford; the National Institute for Health Research (NIHR) Oxford Biomedical Rese… find

Updated 24 October 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2023-survey-large-language-models-medicine/