Elamur Research⌕
☰
home / papers / 2026 / Comparative Evaluation of AI-Generated vs. Expert-written Answer Explanations for a Medical Education Self-Assessment

Comparative Evaluation of AI-Generated vs. Expert-written Answer Explanations for a Medical Education Self-Assessment

Paper · 2026 · Yiyun Zhou, Francis O'Donnell and Victoria Yaneva

Summary

Explanations for medical multiple-choice questions describe why the keyed option is right and why each distractor is wrong. They are useful for learning, but writing them requires clinical expertise and editorial work, so they are often in short supply. Researchers at the National Board of Medical Examiners built a GPT-5 workflow to draft such explanations for a clinical-stage self-assessment.

The workflow has three main steps:

  • Each item is assigned to a General or a Clinical editorial template. An LLM makes the call, with a keyword fallback for uncertain cases.
  • Three exemplars are retrieved from a 50-item reference set, ranked mainly by metadata match and then by lexical overlap.
  • An editor's feedback was used to refine the prompts iteratively, and a parser triggers regeneration whenever a distractor is left unaddressed.

On a separate 50-item set, faculty and students compared AI and expert explanations without knowing which was which. The AI drafts were judged more informative and matched the expert texts on clarity and structure. This suggests LLMs can handle first drafts while experts concentrate on review.

Key points

  • Data: two 50-item MCQ sets. Set 1 was the reference pool for prompts. Set 2 was used for evaluation. Together they cover 49 content areas and 13 competency groups.
  • Retrieval selects top-k = 3 exemplars. Ranking prefers an exact Content Area code match, then a prefix match, then Competency Group and template type.
  • Blinded rating used a balanced incomplete block design in Qualtrics. 8 faculty and 16 students each rated 25 explanation pairs on a three-point scale.
  • Amount of information: a cumulative link mixed model gave OR = 1.99 (95% CI [1.33, 2.99], p = 0.001) in favor of AI. Clarity and structure showed no significant difference, likely because of ceiling effects.
  • Faculty flagged 20% of AI explanations for correction versus 38% of expert ones. However, the AI outputs sometimes contained verbosity or unsupported clinical details.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: National Board of Medical Examiners.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksMedicine · 275 worksHealth Informatics · 214 worksPublic Health, Environmental and Occupational Health · 11 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Yiyun Zhou, Francis O'Donnell and Victoria Yaneva. 2026. Comparative Evaluation of AI-Generated vs. Expert-written Answer Explanations for a Medical Education Self-Assessment. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.31

@inproceedings{zhou2026comparative,
  title = {{Comparative Evaluation of AI-Generated vs. Expert-written Answer Explanations for a Medical Education Self-Assessment}},
  author = {Yiyun Zhou, Francis O'Donnell and Victoria Yaneva},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.31},
  url = {https://doi.org/10.18653/v1/2026.bea-1.31}
}

References

13 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial intelligence in medicine, 155:102938. find
  2. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2025. Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language…
  3. Peter Y Ch’en, Wesley Day, Ryan C Pekson, Juan Barrientos, William B Burton, Allison B Ludwig, Sunit P Jariwala, and Todd Cassese. 2025. Gpt-4 generated answer rationales to multiple choice assessment questions in undergraduate medical education. BMC Medical Education, 25(1):333. find
  4. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-verification reduces hallucination in large language models. In Findings of the association for computational linguistics: ACL 2024, pages 3563–3578.
  5. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  6. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Medexqa: Medical question answering benchmark with multiple explanations. In Proceedings of the 23rd Workshop on biomedical natural language processing, pages 167–181. find
  7. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  8. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36:46534–46594. find
  9. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495.
  10. Chirag Shah. 2025. From prompt engineering to prompt science with humans in the loop. Communications of the ACM, 68(6):54–61. find
  11. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  12. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950. find
  13. Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to retrieve in-context examples for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1752–1767.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-comparative-evaluation-ai-generated-vs/