Elamur Research⌕
☰
home / papers / 2024 / Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

Paper · 2024 · Hyunjae Kim, Hyeon Hwang, Jiwoo Lee et al.

Summary

Commercial LLMs such as GPT-4 handle medical exams well, but sending patient data to closed APIs raises privacy and security problems. Open models that can run on local servers usually lack the multi-step reasoning that clinical questions require. The authors present Meerkat, a family of open medical models with 7B, 8B and 70B parameters, built on Mistral-7B, LLaMA-3-8B and LLaMA-3-70B. Fine-tuning combined chain-of-thought (CoT) explanations for 9.3K MedQA questions, about 78K synthetic CoT question–answer items generated from 18 medical textbooks (released as MedBooks-CoT-18), and general instruction-following and chat data. The small models clearly beat earlier open 7B medical models and GPT-3.5 on six multiple-choice benchmarks. Meerkat-7B is the first 7B model to pass the USMLE threshold, and Meerkat-70B slightly exceeds GPT-4. The work suggests that reasoning traces distilled from textbooks can give compact, locally deployable models much of the capability of large proprietary systems.

Key points

  • Six-benchmark averages: Meerkat-7B scored 62.6% and Meerkat-8B scored 67.3%. GPT-3.5 reached 54.8%, MediTron-7B 51.0% and BioMistral-7B 55.4%. Meerkat-70B beat GPT-4 by 1.3 points on average. The benchmarks were MedQA, USMLE sample test, Medbullets, MedMCQA, MMLU-Medical and JAMA Clinical Challenge.
  • MedQA: Meerkat-7B scored 70.6% as a single model and 77.1% with a choice-shuffling ensemble, against a 60% pass mark. Meerkat-70B scored 82.6% and 85.8%, which is 1.2 and 4.4 points above GPT-4.
  • NEJM case challenges: Meerkat-70B diagnosed 21 of 38 complex cases correctly. NEJM readers averaged 13.8 and GPT-4 scored 21.8.
  • K-QA free-form answers: Completeness was 70.3%, 72.2% and 75.4% for the 7B, 8B and 70B models, compared with 71.4% for GPT-3.5. Factuality was about 89.6–90.0%, below the GPT models at roughly 92%, so hallucination remains an issue.
  • Released artifacts: Weights are published on Hugging Face under dmis-lab (meerkat-7b-v1.0, llama-3-meerkat-8b/70b-v1.0), together with the MedQA CoT data and MedBooks-CoT-18.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations🎓Korea UniversitySeoul · South Korea26 works · 9,490 citations

Also on the byline: Seoul National University Hospital.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi and Jaewoo Kang. 2024. Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks. arXiv:2404.00376.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.00376 · arXiv 2404.00376

@misc{kim2024small,
  title = {{Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks}},
  author = {Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi and Jaewoo Kang},
  year = {2024},
  doi = {10.48550/arxiv.2404.00376},
  eprint = {2404.00376},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.00376}
}

References

53 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thirunavukarasu, A.J., Ting, D.S.J., Elangovan, K., Gutierrez, L., Tan, T.F., Ting, D.S.W.: Large language models in medicine. Nature medicine 29(8), 1930– 1940 (2023) find
  2. Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D.C., et al.: Opportunities and challenges for chatgpt and large language models in biomedicine and health. Briefings in Bioinformatics 25(1), 493 (2024) find
  3. Kung, T.H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al.: Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health 2(2), 0000198 (2023)
  4. Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. Nature 620(7972), 172–180 (2023)
  5. Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al.: Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617 (2023)
  6. Nori, H., King, N., McKinney, S.M., Carignan, D., Horvitz, E.: Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023)
  7. Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B.S., Charney, A.W., Nadkarni, G., Klang, E.: Comparing chatgpt and gpt-4 performance in usmle soft skill assessments. Scientific Reports 13(1), 16492 (2023) find
  8. Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al.: Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024) find
  9. Zakka, C., Shad, R., Chaurasia, A., Dalal, A.R., Kim, J.L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al.: Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1(2), 2300068 (2024) find
  10. Tu, T., Palepu, A., Schaekermann, M., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomasev, N., et al.: Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654 (2024) — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) find
  11. Eriksen, A.V., Möller, S., Ryg, J.: Use of GPT-4 to diagnose complex clinical cases. Massachusetts Medical Society (2023) find
  12. Li, X., Zhang, T.: An exploration on artificial intelligence application: From security, privacy and ethic perspective. In: 2017 IEEE 2nd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), pp. 416–420 (2017). IEEE find
  13. Bartoletti, I.: Ai in healthcare: Ethical and privacy challenges. In: Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019, Poznan, Poland, June 26–29, 2019, Proceedings 17, pp. 7–10 (2019). Springer find
  14. Meskó, B., Topol, E.J.: The imperative for regulatory oversight of large language models (or generative ai) in healthcare. NPJ digital medicine 6(1), 120 (2023) find
  15. OpenAI: Introducing chatgpt (2022) find
  16. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
  17. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  18. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023) find
  19. Google: Gemma: Introducing new state-of-the-art open models (2024) find
  20. Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023)
  21. Chen, Z., Cano, A.H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al.: Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023)
  22. Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., Dufour, R.: Biomistral: A collection of open-source pretrained large language models for medical domains. arXiv preprint arXiv:2402.10373 (2024)
  23. Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al.: Me llama: Foundation large language models for medical applications. arXiv preprint arXiv:2402.12749 (2024)
  24. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
  25. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., Fedus, W.: Emergent abilities of large language models. Transactions on Machine Learning Research (2022)
  26. Tay, Y., Dehghani, M., Tran, V.Q., Garcia, X., Bahri, D., Schuster, T., Zheng, H.S., Houlsby, N., Metzler, D.: Unifying language learning paradigms. arXiv preprint arXiv:2205.05131 (2022)
  27. Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022) find
  28. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11(14), 6421 (2021)
  29. Manes, I., Ronn, N., Cohen, D., Ber, R.I., Horowitz-Kugler, Z., Stanovsky, G.: K-qa: A real-world medical q&a benchmark. arXiv preprint arXiv:2401.14493 (2024) find
  30. Toma, A., Lawler, P.R., Ba, J., Krishnan, R.G., Rubin, B.B., Wang, B.: Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. arXiv preprint arXiv:2305.12031 (2023) find
  31. Chen, H., Fang, Z., Singla, Y., Dredze, M.: Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060 (2024)
  32. Wang, X., Wei, J., Schuurmans, D., Le, Q.V., Chi, E.H., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. In: The Eleventh International Conference on Learning Representations (2023)
  33. Nori, H., Lee, Y.T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al.: Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 (2023) find
  34. Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., Zhang, Y.: Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus 15(6) (2023)
  35. Han, T., Adams, L.C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Löser, A., Truhn, D., Bressem, K.K.: Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247 (2023)
  36. Lin, C.-Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
  37. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019) find
  38. Zhang, M., Press, O., Merrill, W., Liu, A., Smith, N.A.: How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534 (2023)
  39. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, 27730–27744 (2022)
  40. Pal, A., Umapathi, L.K., Sankarasubbu, M.: Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning, pp. 248–260 (2022). PMLR — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  41. Abacha, A.B., Agichtein, E., Pinter, Y., Demner-Fushman, D.: Overview of the medical question answering task at trec 2017 liveqa. In: TREC, pp. 1–12 (2017) find
  42. Abacha, A.B., Mrabet, Y., Sharp, M., Goodwin, T.R., Shooshan, S.E., Demner-Fushman, D.: Bridging the gap between consumers’ medication questions and trusted answers. In: MedInfo, pp. 25–29 (2019) find
  43. Abacha, A.B., Yim, W.-w., Fan, Y., Lin, T.: An empirical study of clinical note generation from doctor-patient encounters. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2283–2294 (2023) find
  44. Zhang, X., Tian, C., Yang, X., Chen, L., Li, Z., Petzold, L.R.: Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558 (2023) — [arXiv:2310.14558](https://arxiv.org/abs/2310.14558) find
  45. Alberti, C., Andor, D., Pitler, E., Devlin, J., Collins, M.: Synthetic QA corpora generation with roundtrip consistency. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6168–6173. Association for Computational Linguistics, Florence, Italy (2019). http…
  46. Sattar, H.A.: Fundamentals of Pathology: Medical Course and Step 1 Review. Pathoma.com, ??? (2020) find
  47. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., Hajishirzi, H.: Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 (2022) find
  48. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford Alpaca: An Instruction-following LLaMA model. GitHub (2023) find
  49. Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35, 16344–16359 (2022) find
  50. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)
  51. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P.W., Iyyer, M., Zettlemoyer, L., Hajishirzi, H.: Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 (2023)
  52. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J.E., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (2023)
  53. Ko, M., Lee, J., Kim, H., Kim, G., Kang, J.: Look at the first sentence: Position bias in question answering. arXiv preprint arXiv:2004.14602 (2020)

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2024-small-language-models-learn-enhanced/