Elamur Research⌕
☰
home / papers / 2026 / Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital

Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital

Paper · 2026 · Nettuno Nadalini, Tarannom Mehri, Anne H Hoekman et al.

Summary

Discharge letters move clinical information between care providers, but writing them takes a large share of physicians' time, and letters often arrive late. The authors ran a prospective mixed-methods pilot in a Dutch academic hospital. They embedded a GPT-based draft generator in the Epic EHR and let 21 residents and 4 physician assistants use it during routine work for nine weeks. They tracked actual use through a funnel: drafts generated, drafts copied, and LLM text still present in the final letter, which they estimated by comparing drafts and sent letters with BLEU. They also surveyed the users and compared documentation and turnaround times with a pre-pilot baseline. Users copied drafts for most admissions, and recognisable LLM text survived in roughly three in ten final letters. Nearly all users believed the tool saved time and wanted to keep using it. The logged time data, however, did not show savings, mainly because the tool was used on longer, more complex stays. The study matters because it reports real deployment behaviour across several departments, not just draft quality on a benchmark.

Key points

  • Over 9 weeks, users made 379 generations for 253 admissions across 10 departments. Drafts were copied 148 times, covering 58.5% of admissions, with about 2.7 generations per copy.
  • Of 227 retrievable final letters, 66 (29.1%) had a BLEU-4 score above 0.2, which the authors treat as reliable evidence of reused LLM text. Another threshold, BLEU-4 above 0.5, was met by 31 letters (13.7%). BLEU-2 overstated reuse at low scores.
  • Adoption differed strongly by department. Surgery showed high reuse, while the emergency department and some others rarely copied drafts.
  • 23 of 25 users answered the survey:
    • 86.9% perceived lower documentation time and 60.9% perceived reduced workload.
    • 91.3% intend to keep using the tool and 78.3% consider it ready for wider rollout.
    • Trust in the output was split.
  • Setup: GPT-4o, later GPT-4.1, on Azure with temperature 0 and no fine-tuning. Prompts were in Dutch and built from EHR notes. Every generated statement carried a reference to its source.
  • Logged documentation time per admission went up when the LLM was used, but those admissions also had longer stays. Overall letter turnaround after introduction fell by about 8.5 hours.

Interesting ideas

  • I1. Adoption is measured as a behavioural funnel (generated → copied → traceable in the final document), not only through opinions, which shows where users drop off.
  • I2. Higher-order n-gram overlap (BLEU-4) with a calibrated cut-off serves as a conservative detector that generated text was really kept. The authors checked it against a manual identical-word ratio and found that bigram overlap inflates low-reuse cases.
  • I3. Every piece of the generated summary is linked to the source note it came from, and the links are removed only when the text is exported.
  • I4. Self-reported time savings disagreed with logged time, because the tool was mostly applied to longer, harder cases. This shows that uncontrolled before/after comparisons can mislead.
  • I5. Users were told to write their own draft before reading the model's, to avoid anchoring on its omissions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation of LLM-teacher or silver outputs could log a similar funnel (proposed → accepted by witnesses → surviving in the final treebank) to see which stages lose value.
  • I2 (back): When scoring the text → graph → text compressor, Mova could set BLEU-4-style thresholds against hand-checked samples, so that weak overlap is not mistaken for faithful regeneration.
  • I3 (answer): Answers could attach, per clause, the frame and sentence they were derived from, keeping explanations visible during review and stripping them for plain output.
  • I4 (learn): Mova's evaluations should compare matched difficulty groups and include negative controls, so that gains are not artefacts of which items a method was applied to.
  • I5 (learn): A student parser could commit to its own analysis before seeing the LLM teacher's output, so that disagreements reveal real gaps instead of copied errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏥University Medical Center GroningenGroningen · Netherlands2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMedicine · 241 worksHealth Informatics · 184 worksHealth Professions · 69 worksHealth Information Management · 50 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Nettuno Nadalini, Tarannom Mehri, Anne H Hoekman, Katerina Kagialari, Job N Doornberg, Tom P van der Laan, Jacobien H F Oosterhoff, Rosanne C Schoonbeek and Charlotte M H H T Bootsma-Robroeks. 2026. Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital. arXiv:2604.19774.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.19774 · arXiv 2604.19774

@misc{nadalini2026phase,
  title = {{Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital}},
  author = {Nettuno Nadalini, Tarannom Mehri, Anne H Hoekman, Katerina Kagialari, Job N Doornberg, Tom P {van der Laan}, Jacobien H F Oosterhoff, Rosanne C Schoonbeek and Charlotte M H H T Bootsma-Robroeks},
  year = {2026},
  doi = {10.48550/arxiv.2604.19774},
  eprint = {2604.19774},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.19774}
}

References

46 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hesselink, G. et al. Improving patient handovers from hospital to primary care: A systematic review. Annals of Internal Medicine vol. 157 Preprint at https://doi.org/10.7326/0003-4819-157-6-201209180-00006 (2012). ↗
  2. Kripalani, S. et al. Deficits in communication and information transfer between hospital-based and primary care physicians: Implications for patient safety and continuity of care. JAMA vol. 297 Preprint at https://doi.org/10.1001/jama.297.8.831 (2007). ↗
  3. Souza, D. L. B. et al. Trends of multimorbidity in 15 European countries: a population-based study in community-dwelling adults aged 50 and over. BMC Public Health 21, (2021). find
  4. King, D. E., Xiang, J. & Pilkerton, C. S. Multimorbidity trends in United States adults, 1988–2014. Journal of the American Board of Family Medicine 31, (2018). find
  5. Westbrook, J. I., Ampt, A., Kearney, L. & Rob, M. I. All in a day’s work: An observational study to quantify how and with whom doctors on hospital wards spend thier time. Medical Journal of Australia 188, (2008). find
  6. Ammenwerth, E. & Spötl, H. P. The time needed for clinical documentation versus direct patient care - A work-sampling analysis of physicians’ activities. Methods Inf. Med. 48, (2009). find
  7. Horwitz, L. I. et al. Comprehensive quality of discharge summaries at an academic medical center. J. Hosp. Med. 8, (2013). find
  8. Ash, J. S., Berg, M. & Coiera, E. Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors. Journal of the American Medical Informatics Association 11, (2004). find
  9. Schulte, F., Fry, E., Shulte, F. & Fortune, E. F. Death By 1,000 Clicks: Where Electronic Health Records Went Wrong. Kaiser Health News (2019). find
  10. Mamykina, L., Vawdrey, D. K. & Hripcsak, G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers. Academic Medicine 91, (2016). find
  11. Joukes, E., Abu-Hanna, A., Cornet, R. & De Keizer, N. F. Time Spent on Dedicated Patient Care and Documentation Tasks before and after the Introduction of a Structured and Standardized Electronic Health Record. Appl. Clin. Inform. 9, (2018). find
  12. Arndt, B. G. et al. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann. Fam. Med. 15, (2017). find
  13. Sinsky, C. et al. Allocation of physician time in ambulatory practice: A time and motion study in 4 specialties. Ann. Intern. Med. 165, (2016). find
  14. Wang, J. K., Ouyang, D., Hom, J., Chi, J. & Chen, J. H. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One 14, (2019). find
  15. Friedberg, M. W. et al. Factors Affecting Physician Professional Satisfaction and Their Implications for Patient Care, Health Systems, and Health Policy. Rand Health Q. 3, (2014). find
  16. Shanafelt, T. D. et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Mayo Clin. Proc. 91, (2016). find
  17. Kossman, S. P. & Scheidenhelm, S. L. Nurses’ perceptions of the impact of electronic health records on work and patient outcomes. CIN - Computers Informatics Nursing 26, (2008). find
  18. Rao, S. K. et al. The impact of administrative burden on academic physicians: Results of a hospital-wide physician survey. Academic Medicine 92, (2017). find
  19. Thirunavukarasu, A. J. et al. Large language models in medicine. Nature Medicine vol. 29 Preprint at https://doi.org/10.1038/s41591-023-02448-8 (2023). ↗
  20. Roberts, K. Large language models for reducing clinicians’ documentation burden. Nat. Med. 30, 942–943 (2024). find
  21. Tripathi, S., Sukumaran, R. & Cook, T. S. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. Journal of the American Medical Informatics Association vol. 31 Preprint at https://doi.org/10.1093/jamia/ocad258 (2024). ↗
  22. Fornasiere, R., Brunello, N., Scotti, V. & Carman, M. J. Medical Information Extraction with Large Language Models. https://aclanthology.org/2024.icnlsp-1.47/ (2024). ↗
  23. Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, (2024).
  24. Schoonbeek, R. C. et al. Quality and efficiency of integrating customised large language model-generated summaries versus physician-written summaries: a validation study. BMJ Open 15, e099301 (2025). find
  25. Small, W. R. et al. Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model. JAMA Netw. Open 8, e2526339 (2025). find
  26. Williams, C. Y. K. et al. Physician- and Large Language Model–Generated Hospital Discharge Summaries. JAMA Intern. Med. 185, 818 (2025). find
  27. Goswami, J., Prajapati, K. K., Saha, A. & Saha, A. K. Parameter-efficient fine-tuning large language model approach for hospital discharge paper summarization. Appl. Soft Comput. 157, 111531 (2024). find
  28. Ando, K., Okumura, T., Komachi, M., Horiguchi, H. & Matsumoto, Y. Is artificial intelligence capable of generating hospital discharge summaries from inpatient records? PLOS Digital Health 1, (2022). find
  29. Chua, C. E. et al. Integration of customised LLM for discharge summary generation in real-world clinical settings: a pilot study on RUSSELL GPT. Lancet Reg. Health West. Pac. 51, 101211 (2024). find
  30. Searle, T., Ibrahim, Z., Teo, J. & Dobson, R. J. B. Discharge summary hospital course summarisation of in patient Electronic Health Record text with clinical concept guided deep pre-trained Transformer models. J. Biomed. Inform. 141, (2023). find
  31. Williams, C. Y. K. et al. Physician- and Large Language Model-Generated Hospital Discharge Summaries. JAMA Intern. Med. 185, 818–825 (2025). find
  32. Challener, D., Ayanian, S., Ryu, A., O’Horo, J. & Heaton, H. Quality assessment of artificial intelligence-generated versus human-written hospital summaries evaluating detail, usefulness, and continuity of care. J. Hosp. Med. https://doi.org/10.1002/jhm.70163 (2025) doi:10.1002/jhm.70163. ↗
  33. van de Sande, D. et al. To warrant clinical adoption AI models require a multifaceted implementation evaluation. NPJ Digit. Med. 7, (2024). find
  34. van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S. & Krahmer, E. Best practices for the human evaluation of automatically generated text. in INLG 2019 - 12th International Conference on Natural Language Generation, Proceedings of the Conference (2019). doi:10.18653/v1/w19-8643. ↗
  35. Bedi, S. et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333, 319–328 (2025). find
  36. Song, J. W., Park, J., Kim, J. H. & You, S. C. Large Language Model Assistant for Emergency Department Discharge Documentation. JAMA Netw. Open 8, e2538427 (2025). find
  37. Lee, S., Song, J. W., You, S. C. & Kim, J. H. Shifts in emergency physicians’ attitudes toward large language model-based documentation: a pre- and post-implementation study. Sci. Rep. 15, 40643 (2025). find
  38. Bootsma-Robroeks, C. M. H. H. T. et al. AI-generated draft replies to patient messages: exploring effects of implementation. Front. Digit. Health 7, 1588143 (2025). find
  39. Choudhury, A., Shahsavar, Y. & Shamszare, H. User Intent to Use DeepSeek for Health Care Purposes and Their Trust in the Large Language Model: Multinational Survey Study. JMIR Hum. Factors 12, e72867–e72867 (2025). find
  40. Blease, C. et al. Generative Artificial Intelligence in Primary Care: Qualitative Study of UK General Practitioners’ Views. J. Med. Internet Res. 27, e74428 (2025). find
  41. Proctor, S., Lawton, G. & Sinha, S. An AI-Powered Strategy for Managing Patient Messaging Load and Reducing Burnout. Appl. Clin. Inform. 16, 747–752 (2025). find
  42. Ke, Y. H. et al. Clinical and economic impact of a large language model in perioperative medicine: a randomized crossover trial. NPJ Digit. Med. 8, 462 (2025). find
  43. Nofal, H. A. et al. The impact of an artificial intelligence enhancement program on healthcare providers’ knowledge, attitudes, and workplace flourishing. Front. Public Health 13, (2025). find
  44. Khan Rony, M. K. et al. Healthcare workers’ knowledge and attitudes regarding artificial intelligence adoption in healthcare: A cross-sectional study. Heliyon 10, e40775 (2024). find
  45. Gallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat. Med. 31, (2025). find
  46. Ng, Z. Q. P., Ling, L. Y. J., Chew, H. S. J. & Lau, Y. The role of artificial intelligence in enhancing clinical nursing care: A scoping review. J. Nurs. Manag. 30, (2022). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nadalini-2026-phase-1-implementation-llm-generated/