Elamur Research
☰
home / papers / 2023 / ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge

ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2303.14070

Summary

General-purpose chat LLMs such as ChatGPT often answer health questions unreliably because they lack targeted clinical training. The authors build ChatDoctor, a medical assistant based on Meta's LLaMA-7B, in two training stages. First, the model learns general dialogue from the 52K Stanford Alpaca instruction set. Second, it is adapted on HealthCareMagic-100k, about 100,000 real patient–doctor exchanges from an online consultation service. These were filtered for length and errors, anonymized, and grammar-corrected with LanguageTool. Roughly 10,000 iCliniq conversations serve as an independent test set.

Beyond fine-tuning, the model gets a retrieval "knowledge brain." It extracts keywords from a query and fetches matching text from Wikipedia or an offline MedlinePlus-based disease database. It then condenses the best-ranked chunks and composes its reply from them. Against ChatGPT on iCliniq answers, the fine-tuned model scores higher in precision, recall and F1, and retrieval lets it handle new topics such as Mpox. Code, data and weights are public, though the authors restrict use to research.

Key points

  • Base model: LLaMA-7B. It was tuned first on Alpaca's 52K instructions, then on HealthCareMagic-100k, using 6× A100 GPUs for about 3 hours (batch 192, 3 epochs, max length 512 tokens, warmup 0.03, no weight decay).
  • Datasets released: HealthCareMagic-100k for training and about 10k iCliniq dialogues for testing, stratified across medical specialties. Both are on GitHub (Kent0n-Li/ChatDoctor) along with code and weights.
  • Retrieval pipeline: the model mines keywords via prompts and uses term-matching search to split sources into equal chunks ranked by keyword hits. It reads the top 5 chunks, summarizes them, and merges the results into a final answer.
  • The offline knowledge base, built from MedlinePlus, covers diseases, symptoms, tests/treatments and medications. It can be updated without retraining the model.
  • The model beat ChatGPT on precision, recall and F1 against iCliniq doctor answers. The authors warn about hallucinations and recommend physician oversight rather than standalone clinical use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 128 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.14070 · arXiv 2303.14070

@misc{li2023chatdoctor,
  title = {{ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge}},
  author = {Li, Yunxiang and Li, Zihan and Zhang, Kai and Dan, Ruilong and Jiang, Steve and Zhang, You},
  year = {2023},
  doi = {10.48550/arxiv.2303.14070},
  eprint = {2303.14070},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.14070}
}

References

6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13.14.15.16.17.18.19. Training language models to follow instructions with human feedback . (2022). Accessed: April 3, 2023: http://arXiv:2203.02155. Self-instruct: aligning language model with self generated instructions . (2022). Accessed: December 20, 2022: htt… ↗
  2. JMIR Med Educ. 2023, 9:45312-2023. Abacha AB, Zweigenbaum P: Means: a medical question-answering system combining NLP techniques and semantic web technologies. find
  3. Inf Process Manag. 2015, 51:570-94. Stanford alpaca: an instruction-following llama model. (2023). Accessed: April 3, 2023: https://github.com/tatsu-lab/stanford\_alpaca. Bertscore: Evaluating text generation with bert . (2020). Accessed: April 21, 2020: http://arXiv:1904.09675. ↗
  4. Gessain A, Nakoune E, Yazdanpanah Y: Monkeypox. N Engl J Med. 2022, 387:1783-93. 10.1056/NEJMra2208860 Beeson AM, Haston J, McCormick DW, Reynolds M, Chatham-Stephens K, McCollum AM, Godfred-Cato S: Mpox in children and adolescents: epidemiology, clinical features, diagnosis, and management . Pediat… find our card
  5. Crit Care. 2023, 27:180. 10.1186/s13054-023-04473-y Beutel G, Geerits E, Kielstein JT: Artificial hallucination: GPT on LSD? . Crit Care. 2023, 27:148. 10.1186/s13054-023-04425-6 Retrieval system evaluation. (2005). Accessed: September 26, 2005: https://www.nist.gov/publications/retrieval-system-eva… ↗
  6. J Med Ethics. 2020, 46:478-81. 10.1136/medethics-2019-105935 — [doi:10.1136/medethics-2019-105935](https://doi.org/10.1136/medethics-2019-105935) find