Elamur Research
☰
home / papers / 2020 / What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2009.13081

Summary

Most open-domain QA benchmarks have one of two weaknesses. Some ask for answer spans that surface word matching can locate. Others test only school-level knowledge. Jin et al. address this with MedQA, a free-form multiple-choice OpenQA dataset taken from professional medical licensing exams in the US, Mainland China and Taiwan. It covers English, simplified Chinese and traditional Chinese. With the questions, the authors release a large corpus of medical textbook text that systems must search for supporting knowledge. Their baselines follow the standard two-stage design: a document retriever gathers evidence passages, then a rule-based or neural reader, including BERT-style pre-trained models, picks an option. Even the strongest system performs poorly. It reaches 36.7% test accuracy on English, 42.0% on traditional Chinese and 70.1% on simplified Chinese. Manual analysis shows retrieval is the main bottleneck, especially for questions that need two reasoning steps, such as diagnosing first and then choosing a treatment. MedQA is therefore a hard, cross-lingual testbed for reasoning that depends on domain knowledge.

Key points

  • Size: 12,723 English (USMLE), 34,251 simplified Chinese (MCMLE) and 14,123 traditional Chinese (TWMLE) questions, 61,097 in total, all manually annotated. Data and code are at github.com/jind11/MedQA.
  • The authors compare it with LiveQA, Medication QA, BioASQ, emrQA, MEDIQA and MedQuAD and present it as the first open-domain medical QA set in multiple-choice format.
  • Evidence check on 100 development samples using the top 25 retrieved paragraphs: no evidence was found for 68% of USMLE questions, compared with 4% for MCMLE and 23.3% for TWMLE.
  • On USMLE, neural readers score below a plain IR baseline because the retrieved evidence is poor. Almost all USMLE items are clinical case questions.
  • Retrieval tends to succeed when distinctive symptoms point to a single diagnosis. It fails when symptoms are common to many conditions or the question needs a second inference step, such as treatment, tests or mechanism.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

476 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 465 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2009.13081 · arXiv 2009.13081

@misc{jin2020what,
  title = {{What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams}},
  author = {Jin, Di and Pan, Eileen and Oufattole, Nassim and Weng, Wei-Hung and Fang, Hanyi and Szolovits, Peter},
  year = {2020},
  doi = {10.48550/arxiv.2009.13081},
  eprint = {2009.13081},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2009.13081}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abacha, A. B.; Agichtein, E.; Pinter, Y.; and Demner-Fushman, D. 2017. Overview of the Medical Question Answering Task at TREC 2017 LiveQA. In TREC. find
  2. Abacha, A. B.; and Demner-Fushman, D. 2019. A question-entailment approach to question answering. BMC bioinformatics 20(1): 511. find
  3. Abacha, A. B.; Mrabet, Y.; Sharp, M.; Goodwin, T. R.; Shooshan, S. E.; and Demner-Fushman, D. 2019. Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers. In MedInfo, 25–29. find
  4. Alsentzer, E.; Murphy, J.; Boag, W.; Weng, W.-H.; Jin, D.; Naumann, T.; and McDermott, M. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, 72– 78. Minneapolis, Minnesota, USA: Association for Computational Linguistics. doi:10… find our card
  5. Ben Abacha, A.; Shivade, C.; and Demner-Fushman, D. 2019. Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, 370–379. Florence, Italy: Association for Computational Linguistics. doi:10.… find
  6. Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5: 135–146. doi:10.1162/tacl a 00051. URL https://www. aclweb.org/anthology/Q17-1010 . find
  7. Chen, D.; Fisch, A.; Weston, J.; and Bordes, A. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1870– 1879. Vancouver, Canada: Association for Computational Linguistics. doi:1… find our card
  8. Clark, C.; and Gardner, M. 2018. Simple and Effective Multi-Paragraph Reading Comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 845–855. Melbourne, Australia: Association for Computational Linguistics. doi:10.18653/v1/P… find
  9. Clark, P.; Etzioni, O.; Khot, T.; Sabharwal, A.; Tafjord, O.; Turney, P.; and Khashabi, D. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Thirtieth AAAI Conference on Artificial Intelligence. find
  10. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Yang, Z.; Wang, S.; and Hu, G. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv preprint arXiv:1906.08101 . find our card
  11. Das, R.; Dhuliawala, S.; Zaheer, M.; and McCallum, A. 2019. Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. ArXiv abs/1905.05733. — [arXiv:1905.05733](https://arxiv.org/abs/1905.05733) find
  12. Demner-Fushman, D.; Mrabet, Y.; and Ben Abacha, A. 2020. Consumer health information and question answering: helping consumers find answers to their health-related information needs. Journal of the American Medical Informatics Association 27(2): 194–201. find
  13. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo… find
  14. Dhingra, B.; Mazaitis, K.; and Cohen, W. W. 2017. Quasar: Datasets for Question Answering by Search and Reading. ArXiv abs/1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
  15. Dunn, M.; Sagun, L.; Higgins, M.; Güney, V. U.; Cirik, V.; and Cho, K. 2017. SearchQA: A New QA Dataset Augmented with Context from a Search Engine. ArXiv abs/1704.05179. find our card
  16. Gao, S.; Agarwal, S.; Jin, D.; Chung, T.; and Hakkani-Tur, D. 2020. From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, 79–89. Online: Association for Computational Linguistics. doi:… find
  17. Gururangan, S.; Marasovi, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL. find
  18. Ha, L. A.; and Yaneva, V. 2019. Automatic Question Answering for Medical MCQs: Can It go Further than Information Retrieval? In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), 418–422. Varna, Bulgaria: INCOMA Ltd. doi:10.26615/978-954-452-0… find
  19. Jin, D.; Gao, S.; Kao, J.-Y.; Chung, T.; and Hakkani-Tür, D. Z. 2019. MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension. ArXiv abs/1910.00458. — [arXiv:1910.00458](https://arxiv.org/abs/1910.00458) find
  20. Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601–1611. Vancouver, Canada: Associa… find our card
  21. Lai, G.; Xie, Q.; Liu, H.; Yang, Y.; and Hovy, E. 2017. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785– 794. Copenhagen, Denmark: Association for Computational Linguistics. doi:10.18653… find our card
  22. Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 . find
  23. Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C. H.; and Kang, J. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics ISSN 1367-4803. doi: 10.1093/bioinformatics/btz682. URL https://doi.org/10.1093/bioinformatics/btz682 . ↗ our card
  24. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv abs/1907.11692. find
  25. Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2381–2391. Brussels, Belgium: Association for Computational L… find
  26. Moen, S.; and Ananiadou, T. S. S. 2013. Distributional semantics resources for biomedical text processing. Proceedings of LBM 39–44. find
  27. Nentidis, A.; Krithara, A.; Bougiatiotis, K.; Krallinger, M.; Rodriguez-Penagos, C.; Villegas, M.; and Paliouras, G. 2020. Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In International Conference of the Cross-Language Evalua… find
  28. Pampari, A.; Raghavan, P.; Liang, J.; and Peng, J. 2018. emrQA: A Large Corpus for Question Answering on Electronic Medical Records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2357–2368. Brussels, Belgium: Association for Computational Linguistics. doi… find
  29. Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019. Language Models as Knowledge Bases? In EMNLP/IJCNLP. find our card
  30. Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 784–789. Melbourne, Australia: Association for Computational Linguistics. doi:1… find our card
  31. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2383–2392. Austin, Texas: Association for Computational Linguistics. doi:10.18653/v1/D16-… find our card
  32. Reddy, S.; Chen, D.; and Manning, C. D. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics 7: 249– 266. find our card
  33. Sun, K.; Yu, D.; Yu, D.; and Cardie, C. 2019. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension. Transactions of the Association for Computational Linguistics 8: 141–155. find
  34. Talmor, A.; Elazar, Y.; Goldberg, Y.; and Berant, J. 2019. oLMpics - On what Language Model Pre-training Captures. ArXiv abs/1912.13283. find our card
  35. Wang, S.; Yu, M.; Guo, X.; Wang, Z.; Klinger, T.; Zhang, W.; Chang, S.; Tesauro, G.; Zhou, B.; and Jiang, J. 2018. R3: Reinforced Ranker-Reader for Open-Domain Question Answering. In AAAI. find
  36. Wang, Z.; Ng, P.; Ma, X.; Nallapati, R.; and Xiang, B. 2019. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. In EMNLP/IJCNLP. find
  37. Yan, M.; Zhang, H.; Jin, D.; and Zhou, J. T. 2020. Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7331–7341. Online: Association for Computational Linguistics. doi:10.18653/v1… find
  38. Yu, W.; Jiang, Z.-H.; Dong, Y.; and Feng, J. 2020. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. ArXiv abs/2002.04326. find our card
  39. Zhang, X.; Wu, J.; He, Z.; Liu, X.; and Su, Y. 2018. Medical exam question answering with large-scale reading comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence. find