Elamur Research⌕
☰
home / papers / 2026 / How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting

How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting

Paper · 2026 · Parker Seegmiller, Joseph Gatto, Sarah E. Greer et al.

Summary

Clinicians are increasingly asked to review LLM-generated drafts of replies to patient portal messages. The authors ask whether such drafts actually reduce their workload or just shift effort into editing. With 13 primary care physicians and nurses, they built a taxonomy of what good clinician replies contain: 8 themes broken into 67 frames. They then created EditJudge, a reference-based LLM-as-judge framework. It estimates how much a clinician would have to delete from or add to a draft, scored as Edit-F1 at two levels: specific content and broader themes. Data come from a US academic hospital's portal. Drafts are conditioned on both the patient message and an EHR summary. The authors tested three local and three commercial LLMs with several adaptation strategies, including thematic prompting, RAG, SFT and DPO. Adapting models around the themes improved drafting by about 33% over 0-shot baselines. Models were weakest at asking patients follow-up questions, and clinician preferences differed widely. The authors conclude that useful drafting will likely require tailoring models to individual clinicians.

Key points

  • Source data: 610k portal messages (2020–2024) yielded 146k patient–clinician conversations from 10,105 adult patients. About 144k conversations were used for training and the rest for evaluation.
  • Three evaluation sets:
    • IPPM: 300 samples, 4 clinicians, theme-guided responses.
    • SyPPM: 100 semi-synthetic samples, publicly released on Hugging Face as PortalPal-AI/Patient-Message-Response-Drafting.
    • SoCPPM: 300 real responses sent in practice by 196 clinicians.
  • Taxonomy: The taxonomy groups reply content into 8 themes: Logistics, Care Coordination and Contingency Planning, alongside Medical Planning, Medical Assessment, Medication Questions, Symptom Questions and Empathy.
  • EditJudge: Its content-level and theme-level Edit-F1 scores work like precision (what must be removed from a draft) and recall (what must be added). The best models would cut clinician edits by only about 25–26%.
  • Privacy: Commercial LLMs were run only on the de-identified SyPPM data. All real patient data stayed on a secure server.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dartmouth CollegeHanover · United States37 works · 42,625 citations

Also on the byline: Dartmouth Health.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Parker Seegmiller, Joseph Gatto, Sarah E. Greer, Ganza Belise Isingizwe, Rohan Ray, Timothy E. Burdick and Sarah Masud Preum. 2026. How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1505

@inproceedings{seegmiller2026how,
  title = {{How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting}},
  author = {Parker Seegmiller, Joseph Gatto, Sarah E. Greer, Ganza Belise Isingizwe, Rohan Ray, Timothy E. Burdick and Sarah Masud Preum},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1505},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1505}
}

References

72 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Anthropic. 2025. Claude 4.5 sonnet. find
  4. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, and 1 others. 2025. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775.
  5. Tita Alissa Bach, Aleksandar Babic, Narae Park, Tor Sporsem, Rasmus Ulfsnes, Henrik Smith-Meyer, and Torkel Skeie. 2024. Using llm-generated draft replies to support human experts in responding to stakeholder inquiries in maritime industry: A real-world case study of industrial ai. arXiv preprint ar… — [arXiv:2412.12732](https://arxiv.org/abs/2412.12732) find
  6. Elizabeth Baltaro, Wendy Henderson, and Karen M Goldstein. 2022. Patient electronic messaging: 12 tips to save time. Family Practice Management, 29(6):5–9. find
  7. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  8. Sally L Baxter, Christopher A Longhurst, Marlene Millen, Amy M Sitapati, and Ming Tai-Seale. 2024. Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned. JAMIA open, 7(2):ooae028. find
  9. Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M. Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey Jain, Leonardo Schettini, Mehr Kashyap, Jason Alan Fries, Akshay Swaminathan, Philip Chung, Fateme Nateghi, Asad Aali, and 62 others. 2025. Medhelm:… find
  10. Joshua M. Biro, Jessica L. Handley, J. Malcolm Mc-Curry, Adam Visconti, Jeffrey M Weinfeld, J. Gregory Trafton, and Raj M. Ratwani. 2025. Opportunities and risks of artificial intelligence in patient portal messaging in primary care. NPJ Digital Medicine, 8. 32629 10 find
  11. Charlotte MHHT Bootsma-Robroeks, Jessica D Workum, Stephanie CE Schuit, Anne Hoekman, Tarannom Mehri, Job N Doornberg, Tom P van der Laan, and Rosanne C Schoonbeek. 2025. Ai-generated draft replies to patient messages: exploring effects of implementation. Frontiers in Digital Health, 7:1588143. find
  12. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77–101. find
  13. Jeffrey Budd. 2023. Burnout related to electronic health record use in primary care. Journal of Primary Care & Community Health, 14. find
  14. Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16051– 1…
  15. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024a. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669.
  16. Shan Chen, Marco Guevara, Shalini Moningi, Frank Hoebers, Hesham Elhalawani, Benjamin H Kann, Fallon E Chipidza, Jonathan Leeman, Hugo JWL Aerts, Timothy Miller, and 1 others. 2024b. The effect of using a large language model to respond to patient messages. The Lancet Digital Health, 6(6):e379– e381… find
  17. Wenyuan Chen, Fateme Nateghi Haredasht, Kameron C Black, Francois Grolleau, Emily Alsentzer, Jonathan H Chen, and Stephen P Ma. 2025. Retrieval-augmented guardrails for ai-drafted patient-portal messages: Error taxonomy construction and large-scale evaluation. arXiv preprint arXiv:2509.22565. — [arXiv:2509.22565](https://arxiv.org/abs/2509.22565) find
  18. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  19. Eden English, Janelle Laughlin, Jeffrey Sippel, Matthew DeCamp, and Chen-Tan Lin. 2024a. Utility of artificial intelligence–generative draft replies to patient messages. JAMA Network Open, 7(10):e2438573– e2438573. find
  20. Eden F. English, Janelle Laughlin, Jeffrey Sippel, Matthew DeCamp, and Chen-Tan Lin. 2024b. Utility of artificial intelligence–generative draft replies to patient messages. JAMA Network Open, 7. find
  21. Jordan Eschler, Leslie S Liu, Lisa M Vizer, Jennifer B McClure, Paula Lozano, Wanda Pratt, and James D Ralston. 2015. Designing asynchronous communication tools for optimization of patient-clinician coordination. In AMIA Annual Symposium Proceedings, volume 2015, page 543. find
  22. Patricia Garcia, Stephen P. Ma, Shreya J. Shah, Margaret Smith, Yejin Jeong, Anna Devon-Sand, Ming Tai-Seale, Kevin Takazawa, Danyelle Clutter, Kyle Vogt, Carlene Lugtu, Matthew Rojo, Steven Lin, Tait Shanafelt, Michael A. Pfeffer, and Christopher Sharp. 2024. Artificial intelligence–generated draft… find
  23. Joseph Gatto, Parker Seegmiller, Timothy Burdick, Philip Resnik, Roshnik Rahat, Sarah DeLozier, and Sarah M Preum. 2026. Medical triage as pairwise ranking: A benchmark for urgency in patient portal messages. arXiv preprint arXiv:2601.13178. — [arXiv:2601.13178](https://arxiv.org/abs/2601.13178) find
  24. Joseph Gatto, Parker Seegmiller, Timothy E. Burdick, Inas S. Khayal, Sarah DeLozier, and Sarah M. Preum. 2025. Follow-up question generation for enhanced patient-provider conversations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)… find
  25. Joseph Gatto, Parker Seegmiller, Timothy E. Burdick, and Sarah M. Preum. 2024. In-context learning for preserving patient privacy: A framework for synthesizing realistic patient portal messages. In Machine Learning for Health (ML4H) Findings. find
  26. Joseph Gatto, Parker Seegmiller, Garrett Johnston, and Sarah Masud Preum. 2022. Identifying the perceived severity of patient-generated telemedical queries regarding covid: Developing and evaluating a transfer learning–based solution. JMIR Medical Informatics, 10(9):e37770. find
  27. Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dligach, and 1 others. 2025. Automating evaluation of ai text generation in healthcare with a large language model (llm)-as-a-judge. medRxiv, pages 2025–04. find
  28. Ariana Genovese, Sahar Borna, Cesar Abraham Gomez-Cabello, Syed Ali Haider, Srinivasagam Prabha, Maissa Trabilsy, Cui Tao, Keith T Aziz, Peter M. Murray, and AJ Forte. 2025. Artificial intelligence for patient support: Assessing retrieval-augmented generation for answering postoperative rhinoplasty… find
  29. Cathal Doyle, Laura Lennox, and Derek Bell. 2013. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ open, 3(1):e001570. find
  30. Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Jordi Bayarri-Planas, Adrian Tormos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin-Torres, Lucia Urcelay-Ganzabal, Marta 32630 11 find
  31. Gonzalez-Mallo, Sergio Alvarez-Napagao, Eduard Ayguadé-Parra, and Ulises Cortés Dario Garcia-Gasulla. 2024. Aloe: A family of fine-tuned open healthcare llms. Preprint, arXiv:2405.01886. — [arXiv:2405.01886](https://arxiv.org/abs/2405.01886) find
  32. Paul KJ Han, Tania D Strout, Caitlin Gutheil, Carl Germann, Brian King, Eirik Ofstad, Pål Gulbrand-sen, and Robert Trowbridge. 2021. How physicians manage medical uncertainty: a qualitative study and conceptual taxonomy. Medical decision making, 41(3):275–291. find
  33. Paul A Harris, Robert Taylor, Robert Thielke, Jonathon Payne, Nathaniel Gonzalez, and Jose G Conde. 2009. Research electronic data capture (redcap)—a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2):37… find
  34. Di Hu, Yawen Guo, Yiliang Zhou, Lidia Flores, and Kai Zheng. 2025. A systematic review of early evidence on generative ai for drafting responses to patient messages. npj Health Systems, 2(1):27. find
  35. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  36. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704.
  37. Thinking Machines John Schulman. 2025. Lora without regret. find
  38. Marina Jovic and Salaheddine Mnasri. 2024. Evaluating ai-generated emails: A comparative efficiency analysis. World Journal of English Language, 14(2). find
  39. Minal S Kale and Deborah Korenstein. 2018. Overdiagnosis in primary care: framing the problem and finding solutions. Bmj, 362. find
  40. Jiyeong Kim, Michael L Chen, Shawheen J Rezaei, April S Liang, Susan M Seav, Sonia Onyeka, Julie J Lee, Shivam C Vedak, David Mui, Rayhan A Lal, and 1 others. 2024. Perspectives on artificial intelligence– generated responses to patient messages. JAMA Network Open, 7(10):e2438535–e2438535. find
  41. Jack Krolik, Herprit Mahal, Feroz Ahmad, Gaurav Trivedi, and Bahador Saket. 2024. Towards leveraging large language models for automated medical q&a evaluation. arXiv preprint arXiv:2409.01941. — [arXiv:2409.01941](https://arxiv.org/abs/2409.01941) find
  42. Elina Laukka, Moona Huhtakangas, Tarja Heponiemi, Sari Kujala, Anu-Marja Kaihlanen, Kia Gluschkoff, and Outi Kanste. 2020. Health care professionals’ experiences of patient-professional communication over patient portals: systematic review of qualitative studies. Journal of Medical Internet Research… find
  43. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods… find
  44. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  45. Shuyue Stella Li, Jimin Mun, Faeze Brahman, Pedram Hosseini, Bryceton G Thomas, Jessica M Sin, Bing Ren, Jonathan S Ilgen, Yulia Tsvetkov, and Maarten Sap. Alfa: Aligning llms to ask good questions a case study in clinical reasoning. In Second Conference on Language Modeling. find
  46. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), pages 47–58.
  47. Siru Liu, Allison B McCoy, Aileen P Wright, Babatunde Carew, Julian Z Genkins, Sean S Huang, Josh F Peterson, Bryan Steitz, and Adam Wright. 2024. Leveraging large language models for generating responses to patient messages—a subjective analysis. Journal of the American Medical Informatics Associat… find
  48. Ivan Lorencin, Nikola Tankovic, and Darko Etinger. 2025. Optimizing healthcare efficiency with local large language models. find
  49. Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems, 160(160). find
  50. Kathryn A. Martinez, Rebecca Schulte, Michael B. Rothberg, Maria C Tang, and Elizabeth R. Pfoh. 2023. Patient portal message volume and time spent on the ehr: an observational study of primary care clinicians. Journal of General Internal Medicine, 39:566 – 572. find
  51. Frederick North, Kristine E Luhman, Eric A Mallmann, Toby J Mallmann, Sidna M. Tulledge-Scheitel, Emily J North, and Jennifer L. Pecina. 2019. A retrospective analysis of provider-to-patient secure messages: How much are they increasing, who is doing the work, and is the work happening after hours?… find
  52. Sarah Masud Preum, Sirajum Munir, Meiyi Ma, Mohammad Samin Yasar, David J Stone, Ronald Williams, Homa Alemzadeh, and John A Stankovic. 2021. A review of cognitive assistants for healthcare: Trends, prospects, and future directions. ACM Computing Surveys (CSUR), 53(6):1–37. 32631 12 find
  53. Shuying Qiao, Paul Denny, and Nasser Giacaman. 2025. Oversight in action: experiences with instructor-moderated llm responses in an online discussion forum. In Proceedings of the 27th Australasian Computing Education Conference, pages 95–104. find
  54. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  55. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  56. Matthew Sakumoto and Aditi U. Joshi. 2023. Digital empathy 2.0: Connecting with patients using the written word. Telehealth and Medicine Today, 8(5). find
  57. Malik Sallam, Nesreen A Salim, Muna Barakat, and Ala’a B Al-Tammemi. 2023. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra j, 3(1):e103. find
  58. Ian A Scott, Anton Van Der Vegt, Paul Lane, Steven McPhail, and Farah Magrabi. 2024. Achieving large-scale clinician adoption of ai-enabled decision support. BMJ Health & Care Informatics, 31(1):e100971. find
  59. Rahul Sharma, Pragnya Ramjee, Kaushik Murali, and Mohit Jain. Editing with ai: How doctors refine llm-generated answers to patient queries. In The Second Workshop on GenAI for Health: Potential, Trust, and Policy Compliance. William R Small, Batia Wiesenfeld, Beatrix Brandfield-Harvey, Zoe Jonassen,… find
  60. John A Stankovic, Meiyi Ma, Sarah Masud Preum, and Homa Alemzadeh. 2021. Challenges and directions for ambient intelligence: A cyber physical systems perspective. In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI), pages 232–241. IEEE. find
  61. Moira Stewart. 1995. Effective physician-patient communication and health outcomes: a review. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne, 152 9:1423–33. find
  62. P TaftiAhmad and 1 others. 2021. Probing patient messages enhanced by natural language processing: a top-down message corpus analysis. Health Data Science. find
  63. Ming Tai-Seale, Sally L Baxter, Florin Vaida, Amanda Walker, Amy Sitapati, Chad Osborne, Joseph Diaz, Nimit Desai, Sophie Webb, Gregory Polston, Teresa Helsten, Erin Gross, Jessica Thackaberry, Ammar Mandvi, Dustin Lillie, Steve Li, Geneen T Gin, Suraj A Achar, Heather Hofflich, and 3 others. 2024.… find
  64. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  65. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  66. Louise Underdahl, Mary Ditri, and Lunthita M Duthely. 2024. Physician burnout: Evidence-based roadmaps to prioritizing and supporting personal wellbeing. Journal of Healthcare Leadership, 16:15 – 27. find
  67. Minghao Wu and Alham Fikri Aji. 2025. Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics.
  68. Qi Yan, Zheng Jiang, Zachary Harbin, Preston H Tolbert, and Mark G Davies. 2021. Exploring the relationship between electronic health records and provider burnout: a systematic review. Journal of the American Medical Informatics Association, 28(5):1009– 1021. find
  69. Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. 2024. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.00732. find
  70. M. Zhao, I. Y. Oh, A. Gupta, S. Cohen-Cutler, K. M. Harmoney, A. M. Lai, and B. A. Sisk. 2025. Automating evaluation of llm-generated responses to patient questions about rare diseases. In medRxiv. find
  71. Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, and Xin Gao. 2023. Skingpt-4: an interactive dermatology diagnostic system with visual large language model. arXiv preprint arXiv:2304.10691. — [arXiv:2304.10691](https://arxiv.org/abs/2304.10691) find
  72. Si Sun, Xiaomu Zhou, Joshua C Denny, Trent S Rosenbloom, and Hua Xu. 2013. Messaging to your doctors: understanding patient-provider communications via a portal system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1739–1748. 32632 13 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seegmiller-2026-how-much-would-clinician-edit/