Elamur Research⌕
☰
home / papers / 2023 / PMC-LLaMA: Towards Building Open-source Language Models for Medicine

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

Summary

General-purpose LLMs handle everyday dialogue well but often give plausible yet wrong answers in medicine because they lack domain knowledge. The strongest medical models, such as MedPALM-2, do not disclose their data or training details. The authors propose an open, reproducible recipe for adapting LLaMA to medicine in two stages.

  1. Knowledge injection. The model is further pretrained on MedC-K, a corpus of biomedical papers and medical textbooks. The data mix deliberately favours textbook content.
  2. Instruction tuning. The model is then tuned on MedC-I, a new instruction set that combines medical QA with rationales, patient–doctor conversations and knowledge-graph prompts. During this stage, loss is computed only on responses.

The resulting 13B-parameter PMC-LLaMA was tested on public medical QA benchmarks. It outperformed LLaMA-2 and even ChatGPT, despite being far smaller. Ablations confirm that each stage contributes to this result. The work matters because it releases the models, code and datasets openly, giving the community a lightweight and transparent baseline for medical LLM research.

Key points

  • MedC-K corpus: about 4.8M biomedical papers selected from S2ORC (81.1M papers) by their PMC IDs, totalling roughly 75B tokens, plus 30K medical textbooks yielding about 4B tokens after cleaning.
  • Training data mix: each batch combines books, papers and general RedPajama-Data text in a 15:4:1 ratio, which emphasises fundamental textbook knowledge.
  • MedC-I instruction dataset: 202M tokens covering medical conversations, rationale-based QA, and knowledge-graph entity descriptions and relations.
  • Evaluation: benchmarks were MedQA, MedMCQA and PubMedQA, where PMC-LLaMA (13B) beat ChatGPT (175B) and LLaMA-2 (70B).
  • Availability: all resources are released at github.com/chaoyi-wu/PMC-LLaMA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.14454 · arXiv 2304.14454

@misc{wu2023pmc,
  title = {{PMC-LLaMA: Towards Building Open-source Language Models for Medicine}},
  author = {Wu, Chaoyi and Lin, Weixiong and Zhang, Xiaoman and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
  year = {2023},
  doi = {10.48550/arxiv.2304.14454},
  eprint = {2304.14454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.14454}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, R.; Dai, A. M.; et al. 2023. PaLM 2 Technical Report. ArXiv, abs/2305.10403. find our card
  2. Chen, T.; et al. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Chiang, W.-L.; Li, Z.; and others. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  3. Chung, H. W.; et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  4. Computer, T. 2023. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset. find
  5. Diao, S.; Pan, R.; et al. 2023. LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. https://optimalscale.github.io/LMFlow/. ↗
  6. Du, Z.; et al. 2021. Glm: General language model pre-training with autoregressive blank infilling. arXiv preprint arXiv:2103.10360. find our card
  7. Han, T.; Adams, L. C.; et al. 2023. MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data. arXiv preprint arXiv:2304.08247. find our card
  8. Iyer, S.; et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. find our card
  9. Jin, D.; Pan, E.; et al. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421. find our card
  10. Jin, Q.; et al. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146. find our card
  11. Kaplan, J.; McCandlish, S.; Henighan, T. J.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Models. ArXiv, abs/2001.08361. find
  12. Kung, T. H.; Cheatham, M.; Medenilla, A.; Sillos, C.; Leon, L. D.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo, J.; and Tseng, V. 2022. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2. find
  13. Lee, J.; et al. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234–1240. find our card
  14. Lindberg, D. A.; Humphreys, B. L.; and McCray, A. T. 1993. The unified medical language system. Yearbook of medical informatics, 2(01): 41–51. find
  15. Lo, K.; et al. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In 58th Annual Meeting of the ACL, 4969– 4983. Online: ACL. find our card
  16. Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  17. Luo, R.; Sun, L.; Xia, Y.; Qin, T.; Zhang, S.; Poon, H.; and Liu, T.-Y. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6): bbac409. find
  18. Nori, H.; King, N.; et al. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. find our card
  19. OpenAI. 2023a. GPT-4 Technical Report. arXiv:2303.08774. find our card
  20. OpenAI. 2023b. OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt/. ↗
  21. Ouyang, L.; et al. 2022. Training language models to follow instructions with human feedback. NIPS, 35: 27730–27744. find
  22. Pal, A.; Umapathi, L. K.; et al. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, 248–260. find our card
  23. PMLR. Peng, B.; et al. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. find our card
  24. Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; et al. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138. find our card
  25. Taori, R.; et al. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/ stanford_alpaca. ↗
  26. Touvron, H.; Lavril, T.; Izacard, G.; Martinet; et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  27. Touvron, H.; Martin, L.; Stone, K. R.; et al. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, abs/2307.09288. find
  28. Yunxiang, L.; et al. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070. find our card
  29. Zhang, K.; et al. 2023. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv preprint arXiv:2305.17100. — [arXiv:2305.17100](https://arxiv.org/abs/2305.17100) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/