Towards Expert-Level Medical Question Answering with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2305.09617

Summary

Answering medical questions as well as a doctor has long been a major goal for AI. The earlier Med-PaLM passed the USMLE-style threshold, but its answers still fell short of clinicians' answers. This Google Research/DeepMind paper presents Med-PaLM 2, which combines three things: the stronger PaLM 2 base model, further tuning on medical question-answering data, and a new prompting technique called ensemble refinement. With ensemble refinement, the model first generates several reasoning chains and then uses them together to produce a final, improved answer. The model is tested on the MultiMedQA suite, which has both multiple-choice and long-form questions, and on two new adversarial question sets that probe safety, bias and health equity. Med-PaLM 2 scores 86.5% on MedQA, a new state of the art, and is close to or above the best prior results on MedMCQA, PubMedQA and MMLU clinical topics. In head-to-head comparisons, physicians often preferred its long-form answers over answers written by other physicians. The work shows that a general-purpose LLM, aligned to medicine and evaluated by humans in detail, is approaching expert level. The authors stress that it still needs validation in real clinical settings.

Key points

• MedQA (USMLE-style, 1273 questions): 86.5% accuracy, more than 19 points above Med-PaLM's 67.2%. For comparison, the paper cites 86.1% for GPT-4-base and 60.2% for GPT 3.5.
• Comparison with physician answers: physicians ranked answers to 1066 consumer health questions drawn from HealthSearchQA, LiveQA and MedicationQA. They preferred Med-PaLM 2 over physician-written answers on 8 of 9 clinical-utility axes (p < 0.001). For example, raters judged its answers closer to medical consensus 72.9% of the time.
• New adversarial datasets: Adversarial (General) has 58 questions and Adversarial (Health equity) has 182, for 240 in total. They cover topics such as drug use, mental health, misinformation, racial bias in clinical calculators, and access to care. Med-PaLM 2 beat Med-PaLM on every axis, and 90.6% of its answers were rated low risk of harm, against 79.4% for Med-PaLM.
• Multiple-choice benchmarks used: MedQA, MedMCQA (4183 questions from Indian medical entrance exams), PubMedQA (500) and six MMLU medical subsets.
• Limitations the authors acknowledge:
  • The rating rubric has not been formally validated.
  • Model answers are longer than physician answers, which may inflate their ratings.
  • Physicians wrote only one answer per question and were given no clinical context.
  • The adversarial sets cover only part of the safety and equity issues.