Elamur Research
☰
home / papers / 2019 / PubMedQA: A Dataset for Biomedical Research Question Answering

PubMedQA: A Dataset for Biomedical Research Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1259.pdf

Summary

Biomedical QA resources have been either small and expert-labeled (BioASQ has under 3k training items) or large but built automatically, with mostly factoid questions that need little reasoning. The authors use a common pattern in PubMed: many articles have a question as their title and a structured abstract whose conclusion answers it. From these articles they build PubMedQA. A system reads the abstract with its conclusion removed and must answer yes, no or maybe. The removed conclusion serves as a "long answer" that can provide extra supervision. The resource has three parts: 1k items labeled by two M.D. candidates, 61.2k unlabeled items, and 211.3k artificial items made by turning declarative titles into questions with heuristic labels. As a baseline, the authors fine-tune BioBERT in several phases: first on the artificial data, then on pseudo-labeled unlabeled data, then on the labeled set. They also add an auxiliary objective that predicts bag-of-words statistics of the conclusion. The best system reaches 68.1% accuracy, compared with 78.0% for a single human and 55.2% for the majority class. The benchmark tests reasoning over quantitative findings, and it is still far from solved.

Key points

  • Subsets: PQA-L has 1k expert-labeled items (55.2% yes, 33.8% no, 11.0% maybe), split into 500 for 10-fold cross-validation and 500 for testing. PQA-U has 61.2k unlabeled items. PQA-A has 211.3k auto-labeled items (92.8% yes, 7.2% no).
  • Source: About 760k PubMed articles have question titles, and about 120k of those have structured abstracts. More than half of such titles can be answered with yes/no/maybe, compared with about 1% in Natural Questions and 6% in HotpotQA.
  • PQA-A construction: Statement titles with an NP-(VBP/VBZ) structure (found with Stanford CoreNLP) are rewritten as questions and labeled by whether they contain a negation. For PQA-U, a rule-based filter that picks out yes/no/maybe questions agrees with an expert annotator over 93% of the time.
  • Results (reasoning-required setting): Multi-phase BioBERT with long-answer supervision gets 68.1% accuracy. A single human gets 78.0%, and the majority baseline gets 55.2%. Other baselines include shallow features, BiLSTM and ESIM with BioELMo.
  • Open problems: About 21% of contexts give numbers without any text describing them. The authors suggest generating the long answer as a richer auxiliary task. The data is available at pubmedqa.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

495 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 474 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1259

@inproceedings{jin2019pubmedqa,
  title = {{PubMedQA: A Dataset for Biomedical Research Question Answering}},
  author = {Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1259},
  url = {https://doi.org/10.18653/v1/d19-1259}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038. find
  2. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… find our card
  3. Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. arXiv preprint arXiv:1805.04871. find
  4. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  5. Roser Morante, Martin Krallinger, Alfonso Valencia, and Walter Daelemans. 2012. Machine reading of biomedical texts about alzheimers disease. In CLEF 2012 Conference and Labs of the Evaluation Forum-Question Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea, pages 1–14. find
  6. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint arXiv:1809.00732. find
  7. Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. 2018. Bioread: A new dataset for biomedical reading comprehension. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  8. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, and Roser Morante. 2013. Qa4mre 2011-2013: Overview of question answering for machine reading evaluation. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 303– 320. Springe… find
  9. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  10. Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. find
  11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  12. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. arXiv preprint arXiv:1809.01494. find
  13. Hiroaki Sakamoto, Yasunori Watanabe, and Masataka Satou. 2011. Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting? Annals of thoracic and cardiovascular surgery, 17(4):376–382. find
  14. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answeri… find
  15. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. find our card