Elamur Research⌕
☰
home / papers / 2026 / Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

Paper · 2026 · Craig Myles, Patrick Schrempf and David Harris-Birtill

Summary

Errors in clinical documentation can delay care or lead to wrong treatment. The MEDEC benchmark has shown that even frontier LLMs struggle to spot such errors, especially on real hospital notes. This paper asks how much automatic prompt optimisation can help, without retraining any model.

The authors apply the Genetic-Pareto (GEPA) optimiser, implemented in DSPy, to binary error detection. GEPA uses a reflector model to iteratively rewrite a baseline prompt (P1 from the MEDEC paper). Prompts are evolved only on the MEDEC MS train and validation splits, then tested on MS-test and on the out-of-distribution UW-test of real de-identified notes. Inference and reflector roles are filled by GPT-5 and six dense Qwen3 models (0.6B–32B) in 28 pairings.

Optimised prompts lift GPT-5 accuracy from 0.669 to 0.785 and Qwen3-32B from 0.578 to 0.690, a new state of the art close to physician performance. The gains largely come from curbing false alarms. This matters for hospitals because the resulting prompts are short and auditable, and they can run on locally hosted open models that keep patient data private.

Key points

  • Data: MEDEC, with MS (exam-style MedQA scenarios with injected errors; 2,189/574/597 train/val/test) and UW (real University of Washington notes; 160 val / 328 test, no training split).
  • With GPT-5, GEPA added 9.5 points on MS-test and 15.2 points on UW-test. MS-test accuracy reached 0.816, beating MediFact (0.737) and a multi-agent debate approach (78.8%) while needing a single model call.
  • Larger Qwen3 models acting as reflectors also improved smaller Qwen3 inference models, offering a privacy-preserving option. The tiniest Qwen3 variants sometimes failed to transfer their gains to UW.
  • Baseline prompts over-flagged errors, e.g. a 76.2% false-positive rate for Qwen3-32B on MS-test. The evolved prompts encouraged conservative judgments and added medical examples, raising specificity at a small cost in sensitivity.
  • Code: github.com/CraigMyles/clinical-note-error-detection (CC BY 4.0). Limitations include dependence on the starting P1 prompt and truncation of Qwen3 reflections at the 32k-token cap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

Cite

Cite as Craig Myles, Patrick Schrempf and David Harris-Birtill. 2026. Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models. Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.healing-1.20

@inproceedings{myles2026importance,
  title = {{Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models}},
  author = {Craig Myles, Patrick Schrempf and David Harris-Birtill},
  booktitle = {Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.healing-1.20},
  url = {https://doi.org/10.18653/v1/2026.healing-1.20}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, and 1 others. 2025. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. find
  2. Anthropic. 2025. Claude sonnet 4.5 system card. Technical report, Anthropic PBC. System card. find
  3. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Fei Xia, and Meliha Yetisgen. 2024. Overview of the MEDIQA-CORR 2024 shared task on medical error detection and correction. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 596–603, Mexico City, Mexico. Association fo… find
  4. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. 2025. MEDEC: A benchmark for medical error detection and correction in clinical notes. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22539–22550, Vienna, Austria. Associatio… find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  7. Kathrin M Cresswell, Sukhmeet S Panesar, Sarah A Salvilla, Andrew Carson-Stevens, Itziar Larizgoitia, Liam J Donaldson, David Bates, Aziz Sheikh, and World Health Organization’s (WHO) Safer Primary Care Expert Working Group. 2013. Global research priorities to better understand the burden of iatroge… find
  8. Rachel Ann Elliott, Elizabeth Camacho, Dina Jankovic, Mark J Sculpher, and Rita Faria. 2021. Economic analysis of the prevalence and clinical and economic burden of medication error in england. BMJ Quality & Safety, 30(2):96–105. find
  9. Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst. 2024. Medical adaptation of large language and vision-language models: Are we making progress? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12143–12170, Miami, Florida, USA. As… find
  10. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  11. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines.
  12. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  13. Abdine Maiga, Anoop Shah, and Emine Yilmaz. 2025. Error detection in medical note through multi agent debate. In Proceedings of the 24th Workshop on Biomedical Language Processing, pages 124–135, Viena, Austria. Association for Computational Linguistics. find
  14. National Academies of Sciences, Engineering, and Medicine. 2015. Improving Diagnosis in Health Care. The National Academies Press, Washington, DC. find
  15. Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naom… — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  16. EYT Wong, L Verlingue, M Aldea, MA Franzoi, R Umeton, S Halabi, N Harbeck, A Indini, A Prelaj, E Romano, and 1 others. 2025. Esmo guidance on the use of large language models in clinical practice (elcap). Annals of Oncology. xAI. 2025. Grok 4 model card. Technical report, xAI. Model card. find
  17. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  18. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, and 1 others. 2025. Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618. find
  19. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. System card. find
  20. Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page…
  21. Nadia Saeed. 2024. MediFact at MEDIQA-CORR 2024: Why AI needs a human touch. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 346–352, Mexico City, Mexico. Association for Computational Linguistics. find
  22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/myles-2026-importance-prompt-optimisation-error-detection/