Elamur Research⌕
☰
home / papers / 2024 / Me LLaMA: Foundation Large Language Models for Medical Applications

Me LLaMA: Foundation Large Language Models for Medical Applications

Paper · 2024 · Qianqian Xie, Qingyu Chen, Aokun Chen et al.

Summary

General-domain open LLMs such as LLaMA often lack specialized medical knowledge. Commercial systems like ChatGPT and GPT-4 are closed and hard to customize. Earlier open medical models seldom combined continual pre-training with instruction tuning, rarely drew on clinical notes and were evaluated mostly on QA. The authors present Me-LLaMA, a family of 13B and 70B models built on LLaMA2. The base models receive further pre-training on a 129B-token mixture of biomedical papers, de-identified clinical notes and general text. Chat variants are then instruction-tuned on roughly 214K samples. Evaluation covers six text-analysis task types across 12 benchmarks, plus diagnosis of complex clinical cases scored by automatic metrics and human reviewers. Me-LLaMA beats LLaMA2 and prior open medical LLMs in most zero-shot and supervised settings. After task-specific tuning, it exceeds ChatGPT on 7 of 8 datasets and GPT-4 on 5 of 8. Its diagnostic quality is close to both commercial systems. The study shows that domain data is decisive and releases reusable medical foundation models under a data use agreement.

Key points

  • The pre-training corpus has 129B tokens and is the largest in the medical domain. It includes over 3M PubMed Central full texts and over 15M PubMed abstracts (from the Pile), MIMIC-III/MIMIC-IV/MIMIC-CXR notes and a RedPajama subset, mixed at a 15:1:4 ratio of biomedical, clinical and general data.
  • The instruction set has 214,595 samples from sources such as Alpaca, Dolly, ShareGPT, MedInstruct, MEDIQA, LiveQA, the UMLS knowledge graph, clinical guidelines and MIMIC.
  • Training the 70B models took more than 100,000 A100 GPU hours. The 13B model matched the 70B one on most datasets.
  • Tasks include QA, NER, relation extraction, classification, summarization and NLI. Clinical diagnosis was tested on 70 NEJM cases.
  • The models are available on PhysioNet under a credentialed DUA, and the code is at GitHub BIDS-Xu-Lab/Me-LLaMA. Limitations include a 4096-token context and possible factual errors or bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🎓University of FloridaGainesville · United States33 works · 364 citations🎓The University of Texas Health Science Center at HoustonHouston · United States5 works · 36 citations

Also on the byline: University of Florida Health.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

Cite

Cite as Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Xinyu Zhou, Lingfei Qian, Huan He, Dennis Shung, Lucila Ohno-Machado, Yonghui Wu, Hua Xu and Jiang Bian. 2024. Me LLaMA: Foundation Large Language Models for Medical Applications. arXiv:2402.12749.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.12749 · arXiv 2402.12749

@misc{xie2024me,
  title = {{Me LLaMA: Foundation Large Language Models for Medical Applications}},
  author = {Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Xinyu Zhou, Lingfei Qian, Huan He, Dennis Shung, Lucila Ohno-Machado, Yonghui Wu, Hua Xu and Jiang Bian},
  year = {2024},
  doi = {10.48550/arxiv.2402.12749},
  eprint = {2402.12749},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12749}
}

References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023).
  2. Wu, C., Zhang, X., Zhang, Y., Wang, Y. & Xie, W. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023).
  3. Han, T. et al. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247 (2023).
  4. https://openai.com/blog/chatgpt \#\# 5. OpenAI. Gpt-4 technical report (2023). 2303.08774. ↗
  5. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  6. Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  7. Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023).
  8. Peng, C. et al. A study of generative large language model for medical research and healthcare. arXiv preprint arXiv:2305.13523 (2023). find
  9. Chen, Z. et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023).
  10. Gema, Aryo, et al. Parameter-efficient fine-tuning of llama for the clinical domain. arXiv preprint arXiv:2307.03042 (2023). find
  11. Li, Y. et al. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus 15 (2023).
  12. Zhang, X. et al. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558 (2023). — [arXiv:2310.14558](https://arxiv.org/abs/2310.14558) find
  13. Gao, L. et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). find
  14. Johnson, A. E. et al. Mimic-iii, a freely accessible critical care database. Scientific data 3, 1–9 (2016).
  15. Johnson, A. et al. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021) (2020). find
  16. Johnson, A. E. et al. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6, 317 (2019). \#\# 18. Computer, T. Redpajama: an open dataset for training large language models (2023). find
  17. Taori, R. et al. Stanford alpaca: An instruction-following https://github.com/tatsu-lab/stanford_alpaca (2023). llama model. ↗
  18. Conover, M. et al. Free dolly: Introducing the world’s first truly open instruction-tuned llm (2023). find
  19. Zheng, L. et al. Judging llm-as-a-judge with mt-bench and chatbot arena (2023). 2306.05685.
  20. Ben Abacha, A., Shivade, C. & Demner-Fushman, D. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In ACL-BioNLP 2019 (2019).
  21. Ben Abacha, A. et al. Bridging the gap between consumers’ medication questions and trusted answers. In MEDINFO 2019 (2019). find
  22. Abacha, A. B., Agichtein, E., Pinter, Y. & Demner-Fushman, D. Overview of the medical question answering task at trec 2017 liveqa. In TREC, 1–12 (2017). find
  23. Yu, B., Li, Y. & Wang, J. Detecting causal language use in science findings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4664– 4674 (2019). find
  24. https://github.com/microsoft/DeepSpeed ↗
  25. Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (2022).
  26. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019).
  27. Zhang, X., Wu, J., He, Z., Liu, X. & Su, Y. Medical exam question answering with large-scale reading comprehension. In Proceedings of the AAAI conference on artificial intelligence, vol. 32 (2018). find
  28. Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, 248–260 (PMLR, 2022). — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  29. Pampari, A., Raghavan, P., Liang, J. & Peng, J. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint arXiv:1809.00732 (2018).
  30. Uzuner, Özlem, et al. "2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text." Journal of the American Medical Informatics Association 18.5 (2011): 552-556. find
  31. Segura-Bedmar, I., Martínez Fernández, P. & Herrero Zazo, M. Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013) (Association for Computational Linguistics, 2013).
  32. Baker, S. et al. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics 32, 432–440 (2016). find
  33. https://www.kaggle.com/code/ritheshsreenivasan/clinical-text-classification ↗
  34. Cohan, A. et al. A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:1804.05685 (2018).
  35. Bastan, M., Surdeanu, M. & Balasubramanian, N. Bionli: Generating a biomedical nli dataset using lexico-semantic constraints for adversarial examples. arXiv preprint arXiv:2210.14814 (2022). find
  36. Romanov, A. & Shivade, C. Lessons from natural language inference in the clinical domain. arXiv preprint arXiv:1808.06752 (2018).
  37. Kanjee, Zahir, Byron Crowe, and Adam Rodman. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. Jama 330.1 (2023): 78-80. find
  38. McDuff, Daniel, et al. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164 (2023). — [arXiv:2312.00164](https://arxiv.org/abs/2312.00164) find
  39. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). find
  40. Lin, CY. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74–81 (2004).
  41. Chen, Qingyu, et al. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv preprint arXiv:2305.16326 (2023). find
  42. Hu, Yan, et al. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association (2024): ocad259. find
  43. Savage, Thomas, et al. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digital Medicine 7.1 (2024): 20. find
  44. Stiennon, N. et al. Learning to summarize with human feedback. Advances Neural Information Processing Systems 33, 3008–3021 (2020). find
  45. Chen, Y. et al. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307 (2023).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2024-me-llama-foundation-large-language/