Elamur Research
☰
home / papers / 2020 / Fact or Fiction: Verifying Scientific Claims

Fact or Fiction: Verifying Scientific Claims

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.609.pdf

Summary

Research output grows quickly, and during crises like COVID-19 it is hard to judge whether a scientific statement holds up. Wadden et al. define a new task, scientific claim verification. Given a claim, a system must find research abstracts that SUPPORTS or REFUTES it and identify the rationale sentences that justify each decision. To study this task, they built SciFact. Experts turned naturally occurring citation sentences (citances) from S2ORC papers into atomic claims. These citances came from papers citing articles in well-regarded biomedical journals. The experts then labeled the cited abstracts and marked the rationales.

As a baseline, the authors present VeriSci, a three-stage pipeline:

  • abstract retrieval,
  • BERT-based rationale selection,
  • label prediction (SUPPORTS, REFUTES or NOINFO).

Domain adaptation to scientific text gives clearly better results than relying only on models trained on Wikipedia (FEVER) or political news claims. A pilot on CORD-19 shows the system can surface useful evidence for real COVID-19 claims. SciFact matters because it is a difficult benchmark that requires background knowledge and reasoning over specialized literature.

Key points

  • Dataset size: SciFact contains 1,409 expert-written claims checked against a corpus of 5,183 abstracts. For each citance, five distractor abstracts are added; these are papers cited elsewhere in the same citing article.
  • Task design: Claims are judged per claim-abstract pair rather than assigned a global truth value. Every SUPPORTS or REFUTES decision needs at least one minimal rationale.
  • COVID-19 case study: A medical student wrote 36 claims, and VeriSci retrieved evidence from CORD-19. The output was judged plausible for 23 of the 36 claims.
  • Error analysis: The authors name five capabilities that models still lack: scientific background knowledge, directionality (increases vs. decreases), numerical and statistical reasoning, cause and effect, and coreference.
  • Availability: Data and code are at github.com/allenai/scifact, with a leaderboard and demo at scifact.apps.allenai.org. SciFact is also included in the ERASER benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

334 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 324 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.609

@inproceedings{wadden2020fact,
  title = {{Fact or Fiction: Verifying Scientific Claims}},
  author = {Wadden, David and Lin, Shanchuan and Lo, Kyle and Wang, Lucy Lu and Zuylen, Madeleine van and Cohan, Arman and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.609},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.609}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramy Baly, Mitra Mohtarami, James Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In NAACL. find
  2. Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluı́s Màrquez i Villodre, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. task 2: Fac… find
  3. Elaine Beller, Justin Clark, Guy Tsafnat, Clive Elliott Adams, Heinz Diehl, Hans Lund, Mourad Ouzzani, Kristina Thayer, James Thomas, Tari Turner, J. S. Xia, Karen A. Robinson, and Paul P Glasziou. 2018. Making progress with the automation of systematic reviews: principles of the international colla… find
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP. find our card
  5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv, abs/2004.05150. find
  6. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in nlp. In EMNLP. find
  7. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In NAACL. find
  8. Arman Cohan, Luca Soldaini, and Nazli Goharian. 2015. Matching citation text and cited spans in biomedical literature: a search-oriented approach. In NAACL. find
  9. Kevin Bretonnel Cohen, Hoa Trang Dang, Anita de Waard, Prabha Yadav, and Lucy Vanderwende. 2014. find
  10. Tac 2014 biomedical summarization track. https://tac.nist.gov/2014/BiomedSumm/ . ↗
  11. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In SemEval. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  13. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020a. Eraser: A benchmark to evaluate rationalized nlp models. In ACL. find
  14. Jay DeYoung, Eric Lehman, Ben Nye, Iain James Marshall, and Byron C. Wallace. 2020b. Evidence inference 2.0: More data, better models. In BioNLP@ACL. find
  15. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In ACL. find
  16. Bradley Efron and Robert Tibshirani. 1993. An introduction to the bootstrap. find
  17. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In NAACL. find
  18. Yu Gu, Robert Tinn, Hao Cheng, M. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. ArXiv, abs/2007.15779. find our card
  19. Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL. find
  20. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL. find
  21. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In CoNLL. find
  22. Kokil Jaidka, Muthu Kumar Chandrasekaran, Devanshu Jain, and Min-Yen Kan. 2017. The cl-scisumm shared task 2017: Results and key insights. In BIRNDL@JCDL. find
  23. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLP. find our card
  24. Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In NAACL. find
  25. Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. In ACL. find
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  27. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In ACL. find our card
  28. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP. find our card
  29. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In ACL Workshop on Language Technologies and Computational Social Science. find our card
  30. Iain James Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. 2017. Automating biomedical evidence synthesis: Robotreviewer. ACL. find
  31. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William. Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Christopher Wilhelm, Boya Xie,… find
  32. Iain James Marshall and Byron C. Wallace. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews, 8. find
  33. Preslav I Nakov, Ariel S Schwartz, and Marti Hearst. 2004. Citances: Citation sentences for semantic analysis of bioscience text. In SIGIR workshop on Search and Discovery in Bioinformatics. find
  34. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, and Byron C. Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL. find
  35. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In ACL. find our card
  36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  37. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In WWW. find
  38. Tal Schuster, Darsh J. Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In EMNLP. find our card
  39. Amir Soleimani, Christof Monz, and Marcel Worring. 2019. Bert for evidence retrieval and claim verification. In European Conference on Information Retrieval. find
  40. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In NAACL. find our card
  41. Guy Tsafnat, Paul P Glasziou, Miew Keen Choong, Adam G. Dunn, Filippo Galgani, and Enrico W. Coiera. 2014. Systematic review automation technologies. Systematic Reviews, 3:74 – 74. find
  42. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artières, Axel-… find