Elamur Research
☰
home / papers / 2023 / MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data

MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2304.08247

Summary

Clinical use of large language models is limited by privacy: the strongest systems run only behind vendor APIs, so patient data would leave the hospital. The authors address this with openly released, locally deployable medical models and the instruction-tuning data behind them. They built Medical Meadow, a collection of more than 160,000 entries. It combines new question–answer data with existing biomedical NLP datasets, all converted into an instruction-following format. The new data came from Anki medical flashcards and WikiDoc pages, both reworded into Q/A pairs with GPT-3.5-Turbo, and from highly upvoted Stack Exchange answers. They then fine-tuned LLaMA 7b and 13b, either fully (following the Alpaca recipe) or more cheaply with LoRA and 8-bit training. Evaluation used zero-shot accuracy on text-only USMLE Step 1–3 self-assessment questions. Fine-tuned models consistently beat the base models, and MedAlpaca 13b performed best. The cheaper LoRA and 8-bit variants lost accuracy. The work offers a reusable recipe and resource for privacy-preserving medical assistants, though the authors caution that confabulation remains a serious risk.

Key points

  • Medical Meadow data sources:
    • Anki flashcards: 33,955 Q/A pairs
    • Stack Exchange: 52,475 pairs from Academia, Biology, Fitness, Health and Bioinformatics, using answers with at least 5 upvotes
    • WikiDoc: 67,704 Living Textbook pairs and 5,942 Patient Information pairs
    • Existing resources: CORD-19, MMLU, MedQA, the Pubmed Causal Benchmark, ChatDoctor forum dialogues and OpenAssistant
  • Full fine-tuning setup: LLaMA 7b and 13b, 5 epochs, cosine learning-rate schedule, effective batch size 256. Learning rate was 2e-5 for 7b and 1e-5 for 13b.
  • Parameter-efficient setup: LoRA with 8-bit matrix multiplication and an 8-bit optimizer, 3 epochs at 2e-5.
  • Results (Step 1 / Step 2 / Step 3):
    • MedAlpaca 13b: 0.473 / 0.477 / 0.602
    • LLaMA 13b: 0.222 / 0.248 / 0.276
    • MedAlpaca 7b: 0.297 / 0.312 / 0.398
    • ChatDoctor (7b): below 0.19 on all steps
  • Trade-offs and release: LoRA and 8-bit training were faster but scored noticeably lower, which the authors attribute partly to limited hyperparameter tuning. All models and datasets are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

117 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.08247 · arXiv 2304.08247

@misc{han2023medalpaca,
  title = {{MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data}},
  author = {Han, Tianyu and Adams, Lisa C. and Papaioannou, Jens-Michalis and Grundmann, Paul and Oberhauser, Tom and Figueroa, Alexei and Loser, Alexander and Truhn, Daniel and Bressem, Keno K.},
  year = {2023},
  doi = {10.48550/arxiv.2304.08247},
  eprint = {2304.08247},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08247}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744, 2022. find
  2. L. C. Adams, D. Truhn, F. Busch, A. Kader, S. M. Niehues, M. R. Makowski, and K. K. Bressem, “Leveraging gpt-4 for post hoc transformation of free-text radiology reports into structured reporting: A multilingual feasibility study,” Radiology, p. 230725, 2023. PMID: 37014240. find
  3. M. Sallam, “The utility of chatgpt as an example of large language models in healthcare education, research and practice: Systematic review on the future perspectives and potential limitations,” medRxiv, 2023. find
  4. P. Lee, S. Bubeck, and J. Petro, “Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine,” New England Journal of Medicine, vol. 388, no. 13, pp. 1233–1239, 2023. find
  5. AI2, CZI, MSR, Georgetown, NIH, and T. W. House, “Covid-19 open research dataset challenge (cord-19),” 2019. Kaggle Challenge. find
  6. D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt, “Aligning ai with shared human values,” Proceedings of the International Conference on Learning Representations (ICLR), 2021. find our card
  7. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multi-task language understanding,” Proceedings of the International Conference on Learning Representations (ICLR), 2021. find
  8. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” arXiv preprint arXiv:2009.13081, 2020. find our card
  9. B. Yu, Y. Li, and J. Wang, “Detecting causal language use in science findings,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), (Hong Kong, China), Association for Comp… find
  10. L. Yunxiang, L. Zihan, Z. Kai, D. Ruilong, and Z. You, “Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge,” arXiv preprint arXiv:2303.14070, 2023. find our card
  11. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model.” https://github.com/tatsu-lab/stanford\_alpaca, 2023. ↗
  12. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021. find
  13. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm.int8(): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339, 2022. find
  14. T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” 9th International Conference on Learning Representations, ICLR, 2022. find our card
  15. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. find
  16. S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, and S. Paul, “Peft: State-of-the-art parameter-efficient fine-tuning methods.” https://github.com/huggingface/peft, 2022. ↗
  17. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. \#\# 6 Appendix MedAlpaca - An Open-Source Collection of… find