Elamur Research
☰
home / papers / 2022 / MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2203.14371

Summary

Automated answering of real medical exam questions has received little research attention. One reason is that existing biomedical multiple-choice collections are small and cover few subjects. The authors address this with MedMCQA, a benchmark of roughly 194k four-option questions. The questions come from Indian postgraduate medical entrance exams (AIIMS PG and NEET PG) and from mock tests and online test series written by medical professionals. Each item has its correct answer, a written explanation, and a label for one of 21 subjects and about 2.4k topics. During cleaning, the authors removed items that depend on images, tables or outside information. They also fixed formatting and spelling and removed duplicates. The splits are based on the exam a question came from, not random sampling, and near-duplicate questions are kept out of the evaluation sets. The baselines are BERT-family encoders, tested with and without retrieved context passages. The best one, PubMedBERT with PubMed passages, reaches only 47% test accuracy, while top human candidates average about 90%. This gap makes MedMCQA a demanding benchmark for medical reasoning and retrieval-augmented QA.

Key points

  • The final dataset has 193,155 questions: about 183K for training (mock and online test series), 6K for development (NEET PG, 2001 onward) and 4K for testing (AIIMS PG, 1991 onward).
  • To prevent leakage, any question with Levenshtein-based similarity above 0.9 to another question was excluded from the dev and test sets.
  • The baselines compared are BERT Base, BioBERT, SciBERT and PubMedBERT, each in three settings: no context, Wikipedia passages and PubMed passages. PubMedBERT scored best in every setting.
  • PubMed passages raised accuracy by about 4–7 points, while Wikipedia context gave almost no gain. This points to a need for domain-specific knowledge sources.
  • The authors manually checked about 100 wrong predictions. The main causes were questions needing multi-hop reasoning (diagnosis and treatment), poor retrieved passages and arithmetic. The data and a leaderboard are at medmcqa.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

379 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 371 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.14371 · arXiv 2203.14371

@misc{pal2022medmcqa,
  title = {{MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering}},
  author = {Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan},
  year = {2022},
  doi = {10.48550/arxiv.2203.14371},
  eprint = {2203.14371},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.14371}
}

References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Chaitanya Shivade, and Dina Yuxian Gu, Robert Tinn, Hao Cheng, Michael R. Demner-Fushman. Overview of the mediqa 2019 Lucas, Naoto Usuyama, Xiaodong Liu, Tris-shared task on textual inference, question entail-tan Naumann, Jianfeng Gao, and Hoifung Poon. find
  2. Domain-specific language model pretraining for biomedical natural language processing. ArXiv, abs/2007.15779, 2022. find our card
  3. Jaewoo Kang. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, February 2020. ISSN 1367-4803. doi: 10.1093/ bioinformatics/btz682. find our card
  4. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. Covid-qa: A to read and comprehend. NIPS’15: Proceedings of question answering dataset for… find
  5. Siva Reddy, Danqi Chen, and Christopher D. Man-Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-ning. Coqa: A conversational question answer-field, Michael Collins, Ankur Parikh, Chris Al-ing challenge. Transactions of the Association for berti, Danielle Epstein, Illia Polosukhin, Matthew Computatio… find
  6. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz and Eduard Hovy. Race: Large-scale reading com-Kaiser, and Illia Polosukhin. Attention is all prehension dataset from examinations. Proceedings you need. I… find
  7. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Curran Associates, Inc., 2017. URL https: Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su. Medical exam question answering with large-scale reading comprehension. In Proceedings of the AAAI Conference on Artifici… find
  8. David Vilares and Carlos Gomez-Rodr. Head-qa: A healthcare dataset for complex reasoning. In HEAD-QA: A Healthcare Dataset for Complex Reasoning, pages 960–966. Association for Computational Linguistics, 7 2019. doi: 10.18653/ v1/P19-1092. URL https://www.aclweb.org/ anthology/P19-1092. ↗
  9. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Simon Šuster and Walter Daelemans. Clicr: A dataset of clinical case reports for machine reading Chaumond, Clement Delangue, Anthony Moi, comprehension. Proceedings of the 2018 Conference Pierric Cistac, Tim Rault, Remi Louf, Morgan of the North Amer… find our card
  10. Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. find
  11. Yi Yang, Wen tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain question answering. In WikiQA: A Challenge Dataset for Open-Domain Question Answering, pages 2013– 2018. Association for Computational Linguistics, 9 2015. doi: 10.18653/v1/D15-1237. URL https: //www.aclweb.org/a… find our card
  12. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. find our card
  13. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. *Figure 8: Distribution of topics per subject & Cumulative* Frequency G… find