Elamur Research⌕
☰
home / papers / 2023 / ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation

ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation

Paper · 2023 · Guangyu Wang, Guoxing Yang, Zongxin Du et al.

Summary

General-purpose large language models often fall short in medicine. They make factual mistakes, reason weakly and lack grounding in real clinical practice. ClinicalGPT addresses this with a Chinese-language model adapted to clinical work using a broad mix of real-world medical data:

  • online Q&A (cMedQA2)
  • knowledge-graph-derived instructions (cMedQA-KG)
  • licensing-exam questions (MEDQA-MCMLE)
  • doctor–patient dialogues (MedDialog)
  • hospital electronic health records (MD-EHR)

Every task is framed as text generation in the T5 style. Training follows an InstructGPT-like pipeline: supervised instruction tuning, a rank-trained reward model, then PPO reinforcement learning with a KL penalty that keeps the policy close to the supervised model. The authors also apply parameter-efficient fine-tuning. Evaluation covers four areas: medical conversation, exams, diagnosis from health records and question answering. ClinicalGPT beats ChatGLM-6B, LLAMA-7B and BLOOM-7B throughout, most clearly on record-based diagnosis. The work offers a practical recipe and an evaluation framework for building domain-specialised clinical assistants.

Key points

  • Training data:
    • cMedQA2 (120k questions, 226k answers)
    • cMedQA-KG (100k pairs built with templates from the cMeKG, xywy-KG and 39Health-KG knowledge graphs)
    • MEDQA-MCMLE (34k multiple-choice exam items)
    • a 100k-dialogue sample of MedDialog
    • MD-EHR (100k multicenter records across seven disease groups)
  • Diagnosis from records (MD-EHR): ClinicalGPT averages 80.9% accuracy, against 60.3% for BLOOM-7B, 40.9% for ChatGLM-6B and 36.6% for LLAMA-7B. It scores highest in Digestive (90.1%) and Urinary (89.9%) and lowest in Respiratory (64.3%).
  • Question answering: On 388 cMedQA2 questions judged by GPT-4, ClinicalGPT wins 89.7% of comparisons against BLOOM-7B, 85.0% against LLAMA-7B and 67.2% against ChatGLM-6B.
  • Reward data: The reward model was trained on 10k annotated cMedQA2 questions, and 4k validation questions were used for the RL stage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing University of Posts and TelecommunicationsBeijing · China78 works · 33,365 citations🔬State Key Laboratory of Networking and Switching TechnologyBeijing · China9 works · 273 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksMedicine · 275 worksHealth Informatics · 214 works

Cite

Cite as Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan and Xiaohu Li. 2023. ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation. arXiv:2306.09968.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.09968 · arXiv 2306.09968

@misc{wang2023clinicalgpt,
  title = {{ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation}},
  author = {Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan and Xiaohu Li},
  year = {2023},
  doi = {10.48550/arxiv.2306.09968},
  eprint = {2306.09968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.09968}
}

References

24 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  2. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 20…
  3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  6. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
  7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  8. Christian Baumgartner. The potential impact of chatgpt in clinical and translational medicine. Clinical and translational medicine, 13(3), 2023. find
  9. Tyler Cowen. The ai revolution in medicine: Gpt-4 and beyond. 2023. find
  10. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023. find
  11. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021. find
  12. Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061–74071, 2018. find
  13. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
  14. Xuehai He, Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, et al. Meddialog: Two large-scale medical dialogue datasets. arXiv preprint arXiv:2004.03329, 2020. — [arXiv:2004.03329](https://arxiv.org/abs/2004.03329) find
  15. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  16. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  17. BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, and Suzana Ilić eta al. Bloom: A 176b-parameter open-access multilingual language model, 2023. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  19. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. find
  20. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
  21. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
  22. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  24. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-clinicalgpt-large-language-models-finetuned/