Elamur Research⌕
☰
home / papers / 2026 / VAIDYA: Validated Agents for Intelligent Diagnosis and Yielded Analysis

VAIDYA: Validated Agents for Intelligent Diagnosis and Yielded Analysis

Paper · 2026 · Kalash Shah, Gautam Bhutani, Rohitaswa Sarbhangia et al.

Summary

Most medical LLM evaluations hand the model a complete case vignette or a multiple-choice item. Real diagnosis is different: a doctor gathers facts step by step through questioning. This paper targets that gap with two benchmarks and an agent framework.

DiagBench draws on patient–doctor dialogues from Fareez et al., plus reformatted DDXPlus and MedQA cases. The model under test must interview an LLM-based Patient Simulator before naming a diagnosis. MedConvBench scores inquiry quality instead: clinicians listed the questions a case truly requires, and models are rated on how many of them they ask.

VAIDYA follows the physician reasoning scheme of Shimizu (2022), with a separately prompted LLM for each stage:

  • a History Taker labels symptoms as systemic or localized;
  • a symptomatological or anatomical specialist then probes further;
  • an etiological specialist narrows candidate causes until Diagnosis Termination.

The framework works with any backbone LLM. It beats plain LLMs and earlier conversational agents, which suggests explicit clinical structure makes AI diagnosis both more accurate and more transparent.

Key points

  • Reported gains: diagnostic accuracy rises by 15% with MedGemma and 21.9% with gpt-5 as the backbone. Generalization was also checked on SCTBench (Script Concordance Tests).
  • Fixed dialogue schedule: round 1 is the History Taker, and round 2 is one specialist (symptomatological for systemic symptoms, anatomical for localized ones). Three etiological rounds follow, for five in total, after which the agent either concludes or requests more information.
  • Ablation on DiagBench: the full VAIDYA (o4-mini) reaches 62.50%. Accuracy falls when the History Taker or the anatomical/symptomatological stage is removed, and also at K = 3 (59.64%) and K = 7 (61.90%).
  • Failure analysis: clinicians reviewed 25 transcripts per scenario with gpt-5 as the backbone. Plain LLMs showed anchoring bias, leaned too heavily on prior medical history and dismissed atypical presentations; VAIDYA reduced these tendencies.
  • Limitations and availability: both benchmarks are text-only (no lab results or imaging), and the gold question sets may be incomplete. Data and code are released at github.com/FractalAIResearchLabs/VAIDYA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMedicine · 241 worksHealth Informatics · 184 worksHealth Professions · 69 worksHealth Information Management · 50 works

Cite

Cite as Kalash Shah, Gautam Bhutani, Rohitaswa Sarbhangia and J Snehan. 2026. VAIDYA: Validated Agents for Intelligent Diagnosis and Yielded Analysis. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.3

@inproceedings{shah2026vaidya,
  title = {{VAIDYA: Validated Agents for Intelligent Diagnosis and Yielded Analysis}},
  author = {Kalash Shah, Gautam Bhutani, Rohitaswa Sarbhangia and J Snehan},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.3},
  url = {https://doi.org/10.18653/v1/2026.gem-main.3}
}

References

24 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health.
  2. Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022. Ddxplus: A new dataset for automatic medical diagnosis. Advances in neural information processing systems, 35:31306– 31318. find
  3. Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. 2024. Cod, towards an interpretable medical agent using chain of diagnosis. find
  4. Faiha Fareez, Tishya Parikh, Christopher Wavell, Saba Shahab, Meghan Chevalier, Scott Good, Isabella De Blasi, Rafik Rhouma, Christopher McMahon, Jean-Paul Lam, et al. 2022. A dataset of simulated patient-physician medical interviews with a focus on respiratory cases. Scientific Data, 9(1):313. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… find
  6. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.
  7. Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. 2024. Mdagents: An adaptive collaboration of llms for medical decision-making. find
  8. Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli… find
  9. Gpt-Thomas Kuriakose and Thomas Kuriakose. 2020. History taking: The most important clinical test. Clinical Insights and Examination Techniques in Ophthalmology, pages 21–29. find
  10. Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, et al. 2024. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957. — [arXiv:2405.02957](https://arxiv.org/abs/2405.02957) find
  11. Liam G McCoy, Rajiv Swamy, Nidhish Sagar, Minjia Wang, James Cao, Stephen Bacchi, Nigel Fong, Nigel CK Tan, Kevin Tan, Thomas A Buckley, et al. 2025. Do language models think like doctors? medRxiv, pages 2025–02. find
  12. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S. Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R. Webster, Ewa Dominowska, Juraj Gottw… — [arXiv:2312.00164](https://arxiv.org/abs/2312.00164) find
  13. Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Viknesh Sounderajah, Jonathan Carlson, Matthew P Lungren, et al. 2025. Sequential diagnosis with language models. arXiv preprint arXiv:2506.22405. — [arXiv:2506.22405](https://arxiv.org/abs/2506.22405) find
  14. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find
  15. Zhoujian Sun, Cheng Luo, Ziyi Liu, and Zhengxing Huang. 2024. Conversational disease diagnosis via external planner-controlled large language models. arXiv preprint arXiv:2404.04292. — [arXiv:2404.04292](https://arxiv.org/abs/2404.04292) find
  16. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering.
  17. Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. 2024. Medagents: Large language models as collaborators for zero-shot medical reasoning.
  18. Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. 2024. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418–1420. find
  19. Mina Valizadeh and Natalie Parde. 2022. The AI doctor is in: A survey of task-oriented dialogue systems for healthcare applications. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6638–6660, Dublin, Ireland. Association for C… find
  20. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  21. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, L… find
  22. Taro Shimizu. 2022. System 2 diagnostic process for the next generation of physicians:“inside” and “outside” brain—the interplay between human and machine. Diagnostics, 12(2):356. find
  23. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8. find
  24. As part of the evaluations outlined in Appendix A and Appendix B, we randomly selected 50 overlapping samples from the DiagBench and Med-ConvBench benchmarks, which were subsequently evaluated by the clinician panel. For this study, we choose the conversations recorded with gpt-5 as the backbone LLM… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2026-vaidya-validated-agents-intelligent-diagnosis/