Elamur Research⌕
☰
home / papers / 2019 / HEAD-QA: A Healthcare Dataset for Complex Reasoning

HEAD-QA: A Healthcare Dataset for Complex Reasoning

Paper · 2019 · David Vilares and Carlos Gómez-Rodríguez

Summary

HEAD-QA addresses a gap in question answering resources. Popular benchmarks such as SQuAD and bAbI can often be solved with surface-level cues, and existing reasoning-oriented datasets are mostly limited to grade-school science. To provide a harder, specialist-level test, Vilares and Gómez-Rodríguez collected multiple-choice questions from the Spanish government's annual exams for specialization posts in public healthcare. Candidates sitting these exams already hold degrees and typically study for a year or more. The questions cover six areas: medicine, nursing, biology, chemistry, psychology and pharmacology. The dataset is in Spanish, and the authors add a machine-translated English version, HEAD-QA-EN, for cross-lingual experiments.

The authors evaluate several kinds of systems over Wikipedia:

  • simple control heuristics;
  • a TF-IDF retrieval baseline built on DrQA's Document Retriever;
  • English neural systems adapted to multiple choice (DrQA, BiDAF, DGEM, Decompatt).

The non-neural retrieval approach performs best. Even so, it stays well below human examinees and under the official pass mark. This makes HEAD-QA a demanding benchmark for knowledge-intensive and multilingual QA.

Key points

  • Size and splits: the dataset has 6,765 questions from exams held from 2013 onward, covering MIR, EIR, BIR, QIR, PIR and FIR. The official supervised split is chronological: 2013–2014 exams for training (2,657), 2015 for development (1,366) and later exams for testing (2,742). Random splitting was avoided so that results stay comparable with official human scores.
  • Images: about 14% of the medicine questions rely on accompanying images. These are included in the release but were not used in the experiments.
  • Translation quality: the English version was produced with the Google API. Human raters checked 60 sampled items, giving adequacy scores of 4.35 and 4.71 and a fluency score of 4, all on 5-point scales. Molecular formulae were the main weakness.
  • Evaluation: systems are scored by accuracy and by the exams' POINTS scheme (+3 for a correct answer, −1 for a wrong one). Cross-lingual IR achieved the best results, averaging about 37% accuracy on the supervised test split.
  • Difficulty and human gap: medicine is the hardest category, which the authors link to its much longer questions and answers. On the 2016 exams, the best system scored far below the top-10 human average and below the pass mark in every area.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidade da CoruñaA Coruña · Spain10 works · 273 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 510 worksQuestion answering · 242 works

Cite

Cite as David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A Healthcare Dataset for Complex Reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1092

@inproceedings{vilares2019head,
  title = {{HEAD-QA: A Healthcare Dataset for Complex Reasoning}},
  author = {David Vilares and Carlos Gómez-Rodríguez},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1092},
  url = {https://doi.org/10.18653/v1/p19-1092}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 5010–5015, Brussels, Belgium. Association for Computational Linguistics. find
  2. Asma Ben Abacha and Demner-Fushman Dina. 2016. Recognizing question entailment for medical question answering. In AMIA Annual Symposium Proceedings, volume 2016, page 310. American Medical Informatics Association. find
  3. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4999–5007. find
  4. Asma Ben Abacha, Duy Dinh, and Yassine Mrabet. 2015. Semantic analysis and automatic corpus construction for entailment recognition in medical texts. In Conference on Artificial Intelligence in Medicine in Europe, pages 238–242. Springer. find
  5. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of AAAI. find
  6. Davide Buscaldi and Paolo Rosso. 2006. Mining knowledge from wikipedia for the question answering task. In Proceedings of the International Conference on Language Resources and Evaluation, pages 727–730. find
  7. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics.
  8. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada. Association for Computational Li…
  9. Souvik Kundu and Hwee Tou Ng. 2018. A nil-aware answer extraction framework for question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4243–4252, Brussels, Belgium. Association for Computational Linguistics. find
  10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. find
  11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Ling…
  12. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, pages 2580–2586. find
  13. Bernardo Magnini, Simone Romagnoli, Alessandro Vallin, Jesús Herrera, Anselmo Penas, Vı́ctor Peinado, Felisa Verdejo, and Maarten de Rijke. 2003. The multiple language question answering track at clef 2003. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 471–486. Sp… find
  14. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701.
  15. Anastasios Nentidis, Anastasia Krithara, Konstantinos Bougiatiotis, Georgios Paliouras, and Ioannis Kakadiaris. 2018. Results of the sixth edition of the bioasq challenge. In Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering, pages… find
  16. Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18). find
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
  18. Eleni Kamateri, Theodora Tsikrika, Spyridon Symeonidis, Stefanos Vrochidis, Wolfgang Minker, and Yiannis Kompatsiaris. 2019. A test collection for passage retrieval evaluation of spanish health-related resources. In European Conference on Information Retrieval, pages 148–154. Springer. find
  19. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255. Association for Computational Linguistics.
  20. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages William Poundstone. 2014. Rock breaks scissors: a practi…
  21. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. pages 784–789.
  22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  23. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
  24. José L Vicedo, Ruben Izquierdo, Fernando Llopis, and Rafael Munoz. 2003. Question answering in spanish. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 541– 548. Springer. find
  25. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
  26. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dcn+: Mixed objective and deep residual coattention for question answering. arXiv preprint arXiv:1711.00106. — [arXiv:1711.00106](https://arxiv.org/abs/1711.00106) find
  27. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Lingu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/