Elamur Research⌕
☰
home / papers / 2026 / Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?

Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?

Paper · 2026 · Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim et al.

Summary

The paper tests whether LLM "doctor" teams diagnose better when their members come from different vendors instead of copies of one model, which may share blind spots. The authors reuse the Multi-Agent Conversation (MAC) framework: three doctor agents take turns proposing ranked differentials and a supervisor (o4-mini by default) gives feedback and issues the final answer. All doctors get identical generic prompts. Three set-ups are compared: a lone model, a single-vendor MAC (three copies of o4-mini, Gemini-2.5-Pro or Claude-4.5-Sonnet) and a mixed-vendor MAC (one agent from each vendor). Evaluation covers rare-disease cases from RareBench (MME, HMS, LIRICAL) and a 2024-only subset of DiagnosisArena chosen to limit contamination. The mixed team gives the best recall and accuracy, and the gains hold under a second, embedding-based judge and with other supervisors. Overlap analysis suggests the agents' differing priors complement one another, while homogeneous teams tend to reinforce errors and sometimes drop correct answers their base model had found. For anyone building clinical assistants, this points to model diversity as a practical design choice.

Key points

  • Data: RareBench subsets MME (40 cases, 17 diseases), HMS (88 cases, 39 diseases) and LIRICAL (370 cases, 252 diseases), plus 165 DiagnosisArena cases from 2024, drawn from a pool of 1,113.
  • Protocol: doctors produce Top-10 lists in round-robin order, with a cap of 13 turns. Scoring uses Recall@1/3/5/10 on RareBench and Top-1/Top-5 accuracy on DiagnosisArena.
  • On combined RareBench, the mixed-vendor MAC reaches 39.31 R@1 and 61.35 R@10, ahead of the strongest single model (Gemini: 37.58 / 56.41) and every single-vendor team.
  • Its lead also holds under a second judge, BioLORD (FRemyCompany/BioLORD-2023), which maps predictions to standardized disease names from OMIM, Orphanet and CCRD. The primary judge is o4-mini.
  • In the supervisor ablation on MME, swapping in Gemini or Claude as supervisor leaves the mixed team at 40.0–42.5% R@1, which is 7.5–15.0 points above the best homogeneous team.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Harvard UniversityCambridge · United States126 works · 45,130 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksMedicine · 275 worksHealth Informatics · 214 worksHealth Professions · 75 worksHealth Information Management · 56 works

Cite

Cite as Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim and Pranav Rajpurkar. 2026. Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?. Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.healing-1.1

@inproceedings{yuan2026do,
  title = {{Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?}},
  author = {Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim and Pranav Rajpurkar},
  booktitle = {Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.healing-1.1},
  url = {https://doi.org/10.18653/v1/2026.healing-1.1}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. X. Li and et al. 2025. The alignment paradox of medical large language models in infertility care. arXiv preprint arXiv:2511.00924. — [arXiv:2511.00924](https://arxiv.org/abs/2511.00924) find
  2. Harsha Nori and 1 others. 2025. agnosis with language models. arXiv:2506.22405. — [arXiv:2506.22405](https://arxiv.org/abs/2506.22405) find
  3. Sequential di-arXiv preprint OpenAI. 2025. Openai o4-mini system card. https://openai.com/index/o4-mini-system-card. Accessed: 2025-12-10. ↗
  4. Ana Rath, Annie Olry, Ferdinand Dhombres, Marc M Brandt, Bruno Urbero, and Ségolène Ayme. 2012. Representation of rare diseases in health information systems: the orphanet approach to serve a wide range of end users. Human Mutation, 33(5):803–808. find
  5. François Remy, Kris Demuynck, and Thomas Demeester. 2024. Biolord-2023: Semantic textual representations fusing large language models and clinical knowledge graph insights. Journal of the American Medical Informatics Association, 31(9):1844–1855. find
  6. Khaled Saab, Tao Tu, and 1 others. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416.
  7. Tieliu Shi and Xiaoxia Lin. 2024. Catalogue of chinese rare diseases (ccrd). http://www.unimd.org/ ccrd/. Accessed: 2025-12-19. ↗
  8. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, 31:943–950. find
  9. Akinori Sonoda, Yuka Horiuchi, Takayuki Abe, Genta Shirota, Tomohiro Mikami, Shuhei Nakao, Kengo Yoshii, Misaki Sone, Kazuma Ryuzaki, Saki Kageyama, and 1 others. 2024. Structured clinical reasoning prompt enhances LLM’s diagnostic capabilities in diagnosis please quiz cases. Japanese Journal of Rad… find
  10. Michael J. Sorich, Arduino A. Mangoni, Stephen Bacchi, Bradley D. Menz, and Ashley M. Hopkins. 2024. The triage and diagnostic accuracy of frontier large language models: Updated comparison to physician performance. Journal of Medical Internet Research, 26:e67409. find
  11. H. Yang, M. Li, and et al. 2024. One llm is not enough: Harnessing the power of ensemble learning for medical question answering. medRxiv. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-do-mixed-vendor-multi-agent/