Elamur Research⌕
☰
home / papers / 2022 / SciFact-Open: Towards open-domain scientific claim verification

SciFact-Open: Towards open-domain scientific claim verification

Paper · 2022 · David Wadden, Kyle Lo, Bailey Kuehl et al.

Summary

Current scientific claim verification systems score very well on small benchmarks, but nobody had checked how they behave against literature collections of realistic size. Wadden et al. build SciFact-Open, a test collection where claims must be checked against 500K biomedical abstracts sampled from S2ORC. Annotating every claim–abstract pair at this scale is impossible, so the authors borrow TREC-style pooling from information retrieval. Four strong verification models each rank candidate pairs by confidence. The union of their top predictions is then labeled by experts, and every label is reviewed by a second annotator. Pairs left unannotated are assumed to carry no evidence. Systems built on the original roughly 5K-abstract SciFact lose at least 15 F1 points in this setting, so they are not ready for large-scale use. The new evidence also shows real-world complications, such as studies that contradict each other and evidence that covers only a narrower case than the claim. The pooling approach may help other NLP tasks where full annotation is too costly.

Key points

  • Claims and corpus: SciFact-Open has 279 claims, taken from the SciFact test set after dropping 21 whose source citations lacked metadata.
    • The corpus is a random sample of 500K abstracts from about 6.5M S2ORC medicine/biology papers that have both incoming and outgoing citations.
    • It includes the 5K original SciFact abstracts and applies no quality or journal-impact filter.
  • Pooled models: VERT5ERINI, ParagraphJoint, MultiVerS and MultiVerS10.
    • Candidates are retrieved with BM25 plus a neural re-ranker.
    • A model's confidence that a pair contains evidence is the higher of its SUPPORTS and REFUTES softmax scores.
  • Evaluation: abstract-level, label-only F1; rationales are not required.
    • The negative sampling ratio used in training strongly affects how well models generalize.
  • Observed phenomena: some claims are highly studied and have many evidence abstracts, sometimes conflicting.
    • Evidence is often more specific than the claim it addresses.
    • The authors release 91 claim revisions to support research on this issue.
  • Limitations: few systems were pooled compared with TREC, and only one retrieval system was used.
    • Sensitivity analyses suggest the collection still separates models reliably.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

Cite

Cite as David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang and Hannaneh Hajishirzi. 2022. SciFact-Open: Towards open-domain scientific claim verification. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.347

@inproceedings{wadden2022scifact,
  title = {{SciFact-Open: Towards open-domain scientific claim verification}},
  author = {David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang and Hannaneh Hajishirzi},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.347},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.347}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. ArXiv, abs/2004.05150. our card
  2. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An Empirical Investigation of Statistical Significance in NLP. In EMNLP. find
  3. Daniel Fernando Campos, T. Nguyen, M. Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, L. Deng, and Bhaskar Mitra. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In NeurIPS. our card
  4. Gordon V. Cormack, Christopher R. Palmer, and Charles L. A. Clarke. 1998. Efficient construction of large test collections. In SIGIR. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL. our card
  6. Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021. MS^2: Multi-Document Summarization of Medical Studies. In EMNLP. find
  7. T. Diggelmann, Jordan L. Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. In Tackling Climate Change with ML workshop @ NeurIPS. our card
  8. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. In ACL. find
  9. V. Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In EMNLP. our card
  10. Neema Kotonya and F. Toni. 2020. Explainable Automated Fact-Checking for Public Health Claims. In EMNLP. our card
  11. Xiangci Li, G. Burns, and Nanyun Peng. 2021. A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification. In Workshop on Scientific Document Understanding @ AAAI. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692. find
  13. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, and Daniel S. Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In ACL. our card
  14. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. On Faithfulness and Factuality in Abstractive Summarization. In ACL. find
  15. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering Ambiguous Open-domain Questions. In EMNLP. find
  16. Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021. Scientific Claim Verification with VerT5erini. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis @EACL. find
  17. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR. our card
  18. Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, and William R. Hersh. 2020a. TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19. Journal of the American Medical Informatics Association. find
  19. Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, and William R. Hersh. 2016. Overview of the TREC 2016 clinical decision support track. TREC. find
  20. Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, William R. Hersh, Steven Bedrick, Alexander J. Lazar, and Shubham Pant. 2020b. Overview of the TREC 2020 Precision Medicine Track. TREC. find
  21. S. Robertson and H. Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. find
  22. Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic. In ACL. our card
  23. Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021. Evidence-based Fact-Checking of Health-related Claims. In EMNLP Findings. our card
  24. Tal Schuster, Adam Fisch, and R. Barzilay. 2021. Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence. In NAACL. our card
  25. Darsh J. Shah, Tal Schuster, and R. Barzilay. 2020. Automatic Fact-guided Sentence Modification. In AAAI. find
  26. Karen Sparck Jones and C. J. van Rijsbergen. 1975. Report on the need for and provision of an "ideal" information retrieval test collection. In British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge. find
  27. Dominik Stammbach, Boyang Zhang, and Elliott Ash. 2021. The Choice of Knowledge Base in Automated Claim Checking. ArXiv, abs/2111.07795. — [arXiv:2111.07795](https://arxiv.org/abs/2111.07795) find
  28. Mujeen Sung, Minbyul Jeong, Yonghwa Choi, Donghyeon Kim, Jinhyuk Lee, and Jaewoo Kang. 2022. BERN2: an advanced neural biomedical named entity recognition and normalization tool. ArXiv, abs/2201.02080. find
  29. Nandan Thakur, N. Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In NeurIPS. our card
  30. James Thorne and Andreas Vlachos. 2021. Evidence-based factual error correction. In ACL. find
  31. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for Fact Extraction and VERification. In NAACL. our card
  32. Ellen M. Voorhees and Donna Harman. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT press. find
  33. David Wadden and Kyle Lo. 2021. Overview and Insights from the SciVer Shared Task on Scientific Claim Verification. In Proceedings of the Second Workshop on Scholarly Document Processing @ NAACL. find
  34. David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. MultiVerS: Improving scientific claim verification with weak supervision and full-document context. In NAACL Findings. find
  35. David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In EMNLP. our card
  36. Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and Iain James Marshall. 2021. Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization. In AMIA Joint Summits on Translational Science proceedings. find
  37. Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating Scientific Claims for Zero-Shot Scientific Fact Checking. In ACL. find
  38. Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021. Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification. In EMNLP. find
  39. Justin Zobel. 1998. How reliable are the results of large-scale information retrieval experiments? In SIGIR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wadden-2022-scifact-open-open-domain-scientific/