Elamur Research⌕
☰
home / papers / 2026 / Evaluating LLM Workflows for Generating Clinical Communication Assessment Items: A Comparative Study with Subject-Matter Experts

Evaluating LLM Workflows for Generating Clinical Communication Assessment Items: A Comparative Study with Subject-Matter Experts

Paper · 2026 · Christopher Runyon, Peter Baldwin, Ian Micir et al.

Summary

Generative models are often used to draft items for language-focused tests, but their usefulness in knowledge-heavy fields such as medicine is less clear. This study asks whether GPT5-based pipelines can produce practice items for NBME's Communication Learning Assessment (CLA) tool. In CLA, learners respond aloud to virtual patients. Each item bundles a clinical vignette, learning objectives (LOs), patient-centered behaviors (PCBs), and two exemplar responses.

The authors built four fully automated workflows:

  • Constrained linear (Method 1): a step-by-step build with quality gates.
  • Exploratory branching (Method 2): multi-agent candidate generation followed by critique-driven convergence.

Each workflow was seeded either with only the diagnosis and communication task from a retired licensure MCQ ("a") or with the full MCQ text as well ("b"). Ten trained experts rated 80 items on six rubric dimensions, and the ratings were analyzed with variance decomposition.

The linear approach scored higher, especially on medical accuracy and on alignment with LOs and PCBs. Full MCQ anchoring did not raise accuracy. Few items were usable without edits, and no item was approved by every rater. The study shows that workflow design matters and that expert oversight is still required.

Key points

  • The 2×2 design crossed linear vs. branching workflows with partial vs. full MCQ anchoring. Each of the four methods produced 20 items, for 80 in total, and all were generated with GPT5 on private deployments.
  • Ten subject-matter experts from NBME's Assessment Alliance network scored the items on six quality dimensions using custom rubrics. Raters could skip later dimensions once an item was judged medically unsalvageable.
  • The constrained linear workflow beat exploratory branching, most clearly on medical accuracy and on LO/PCB alignment.
  • Giving the model the full MCQ text brought no measurable accuracy gain. The authors present this as an absence of evidence rather than proof that anchoring cannot help.
  • Variance decomposition into method, item, rater, and residual components was used instead of kappa- or alpha-type agreement indices. Stereotype flags were too rare for comparisons across methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Christopher Runyon, Peter Baldwin, Ian Micir, Kevin Frome, Stephanie Mann, Saed Rezayi, Keelan Evanini and Victoria Yaneva. 2026. Evaluating LLM Workflows for Generating Clinical Communication Assessment Items: A Comparative Study with Subject-Matter Experts. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.12

@inproceedings{runyon2026evaluating,
  title = {{Evaluating LLM Workflows for Generating Clinical Communication Assessment Items: A Comparative Study with Subject-Matter Experts}},
  author = {Christopher Runyon, Peter Baldwin, Ian Micir, Kevin Frome, Stephanie Mann, Saed Rezayi, Keelan Evanini and Victoria Yaneva},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.12},
  url = {https://doi.org/10.18653/v1/2026.bea-1.12}
}

References

23 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yaara Artsi, Vera Sorin, Eli Konen, Benjamin S. Glicksberg, Girish Nadkarni, and Eyal Klang. 2024. Large language models for generating medical examinations: systematic review. BMC Medical Education, 24:354. find
  2. Yigal Attali, Andrew Runge, Geoffrey T LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A Von Davier. 2022. The interactive reading task: Transformer-based automatic item generation. Frontiers in Artificial Intelligence, 5:903077. find
  3. William CM Belzak, Ben Naismith, and Jill Burstein. 2023. Ensuring fairness of human-and ai-generated test items. In International conference on artificial intelligence in education, pages 701–707. Springer. find
  4. Sirine Bouguettaya, Francesco Pupo, Min Chen, and Giancarlo Fortino. 2025. A meta-survey of generative ai in education: Trends, challenges, and research directions. Big Data and Cognitive Computing, 9(9). find
  5. Emilia Brügge, Sarah Ricchizzi, Malin Arenbeck, Marius Niklas Keller, Lina Schur, Walter Stummer, Markus Holling, Max Hao Lu, and Dogus Darici. 2024. Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled… find
  6. Rune Haubo B Christensen. 2018. Cumulative link models for ordinal regression with the r package ordinal. Submitted in J. Stat. Software, 35:1–46. find
  7. Rune Haubo Bojesen Christensen. 2019. ordinal: Regression Models for Ordinal Data. R package version 2019.12-10. find
  8. Michael D. Cusimano, R. Cohen, W. Tucker, J. Murnaghan, R. Kodama, and R. Reznick. 1994. A comparative analysis of the costs of administration of an OSCE (objective structured clinical examination). Academic Medicine, 69(7):571–576. find
  9. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  10. Kilem L. Gwet. 2002. Inter-rater reliability: Dependency on trait prevalence and marginal homogeneity. Statistical Methods for Inter-Rater Reliability Assessment, 2:1–9. find
  11. Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77–89. find
  12. Donald Hedeker and Robert D. Gibbons. 2006. Longitudinal Data Analysis. Wiley, Hoboken, NJ. and Resources for People with REAding DIfficulties (READI)@ LREC-COLING 2024, pages 22–37. find
  13. Tjorven Stamer, Jost Steinhäuser, and Kristina Flägel. 2023. Artificial intelligence supporting the training of communication skills in the education of health care professions: scoping review. Journal of Medical Internet Research, 25:e43311. find
  14. Masaki Uto, Yuto Tomikawa, and Ayaka Suzuki. 2023. Difficulty-controllable neural question generation for reading comprehension using item response theory. In Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023), pages 119–129. find
  15. Imran Zafar, Hadeel Aboueisha, Ibrahim Elhassan, Fouzia Shersad, Suleyman Ayhan Caliskan, Asma Fatima Syeda, Mohammed Al-Houqani, and Mohi El-din Magzoub. 2026. Ten tips for utilizing AI to generate high quality OSCE stations in medical education. Frontiers in Medicine, 12:1744657. find
  16. Yavuz Selim Kıyak and Emre Emekli. 2024. ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgraduate Medical Journal, 100(1189):858–865. find
  17. Geoff LaFlair, Kevin Yancey, Burr Settles, and Alina A von Davier. 2023. Computational psychometrics for digital-first assessments: a blend of ml and psychometrics for item generation and scoring. In Advancing natural language processing in educational assessment, pages 107–123. Routledge. find
  18. Zhiqing Lin and Huilin Chen. 2024. Investigating the capability of chatgpt for generating multiple-choice reading comprehension items. System, 123:103344. find
  19. Kheizaran Miri, Tahere Sarboozi Hoseinabadi, Ali Yaghobi, Sadaf Kholosi, and Mehdi Miri. 2024. Cost management analysis of objective structured clinical examination (OSCE): guide to the universities of medical sciences. BMC Medical Education, 24:1241. find
  20. Sanghamitra M. Misra and Srinivasan Suresh. 2024. Artificial intelligence and objective structured clinical examinations: Using ChatGPT to revolutionize clinical skills assessment in medical education. Journal of Medical Education and Curricular Development, 11:1–6. find
  21. OpenAI. 2025. ChatGPT-5. https://chat.openai.com/. Large language model. ↗
  22. Adriana Quiroga Velasquez. 2026. Medical education case generator GPT workflow. Association of American Medical Colleges (AAMC), Advancing AI Resource Collection. Last updated February 2026. find
  23. Andreas Säuberli and Simon Clematide. 2024. Automatic generation and evaluation of reading comprehension test items with large language models. In Proceedings of the 3rd Workshop on Tools find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/runyon-2026-evaluating-llm-workflows-generating-clinical/