Elamur Research⌕
☰
home / papers / 2023 / Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

Summary

Clinicians lose a large share of their working time condensing electronic health record content into reports and notes. This paper asks whether adapted LLMs can do this summarization at least as well as human experts. The authors tested eight models, from FLAN-T5 and FLAN-UL2 through Llama-2-based variants (Alpaca, Med-Alpaca, Vicuna) to the proprietary GPT-3.5 and GPT-4. They adapted each model with in-context learning or QLoRA fine-tuning. The benchmark covered four tasks drawn from six public datasets: radiology report impressions, patient health questions, ICU progress notes, and doctor–patient dialogue. Outputs were scored with syntactic, semantic and conceptual metrics, including the medical-concept metric MEDCON.

The strongest setup, GPT-4 with in-context examples, then went to a blinded reader study with ten physicians. Readers usually judged its summaries as good as or better than expert-written ones on completeness, correctness and conciseness. A safety analysis linked the errors of both machines and experts to potential harm and classified fabricated content. The results suggest LLM-drafted summaries could reduce documentation burden in clinical practice.

Key points

  • The benchmark spans eight LLMs ranging from 2.7B to 175B parameters, with context windows from 512 to 32,768 tokens. Each model was paired with in-context learning or QLoRA and evaluated on four tasks across six datasets.
  • The datasets are MIMIC-CXR, Open-i, MIMIC-III, MeQSum, ProbSum and ACI-Bench.
  • Larger, newer or medically specialized models did not always win, so the authors report trade-offs between models and adaptation methods.
  • In the reader study (five radiologists and five hospitalists), the best adapted LLM's summaries were rated equivalent to expert summaries in 45% of cases and better in 36%.
  • Code and preprocessed versions of the datasets that do not require PhysioNet access are available at github.com/StanfordMIMI/clin-summ.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

67 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 64 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41591-024-02855-5 · arXiv 2309.07430

@misc{veen2023adapted,
  title = {{Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization}},
  author = {Veen, Dave Van and Uden, Cara Van and Blankemeier, Louis and Delbrouck, Jean-Benoit and Aali, Asad and Bluethgen, Christian and Pareek, Anuj and Polacin, Malgorzata and Reis, Eduardo Pontes and Seehofnerova, Anna and Rohatgi, Nidhi and Hosamani, Poonam and Collins, William and Ahuja, Neera and Langlotz, Curtis P. and Hom, Jason and Gatidis, Sergios and Pauly, John and Chaudhari, Akshay S.},
  year = {2023},
  doi = {10.1038/s41591-024-02855-5},
  eprint = {2309.07430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41591-024-02855-5}
}

References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30. 31. Golob Jr, J. F., Como, J. J. & Claridge, J. A. The painful truth: The documentation burden of a trauma surgeon. Journal of Trauma and Acute Care Surgery 80, 742–747 (2016). Arndt, B. G., Beasley, J. W., Wa… — [arXiv:2308.14089](https://arxiv.org/abs/2308.14089) find
  2. MEDINFO 2019: Health and Wellbeing e-Networks for All, 1194–1198 (2019). Ratwani, R. M., Savage, E., Will, A., Arnold, R., Khairat, S., Miller, K., Fairbanks, R. J., Hodgkins, M. & Hettinger, A. Z. A usability and safety analysis of electronic health records: a multi-center study. Journal of the Ame… find
  3. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of… find
  4. Peng, A., Wu, M., Allard, J., Kilpatrick, L. & Heidel, S. GPT-3.5: Turbo, Fine-Tuning, and API Updates https : //openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates. Accessed: August 22, 2023. 2023. Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S.,… find
  5. Gao, Y., Miller, T., Afshar, M. & Dligach, D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization in Proceedings of the 22nd Workshop on Biomedical Language Processing (2023). Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moo… find
  6. OpenAI, 2023. Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H. & Rush, A. M. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE transactions on visualization and computer graphics 29, 1146–1156 (2022). Wang, J., Shi, E., Yu, S.… find
  7. Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 (2023). Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S. & Ré, C. Hyena hi… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/veen-2023-adapted-large-language-models-outperform/