Elamur Research⌕
☰
home / papers / 2021 / Evidence-based Fact-Checking of Health-related Claims

Evidence-based Fact-Checking of Health-related Claims

Paper · 2021 · Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet et al.

Summary

Most evidence-based fact-checking resources rely on artificially constructed claims, such as edited Wikipedia sentences or rewritten citation sentences. Models trained on them may not handle the vague, hedged and multi-part statements people actually post online, especially about health. The authors present HealthVER, a COVID-19 dataset pairing real-world web claims with evidence from scientific literature.

The dataset was built in three steps:

  • 80 popular pandemic questions were sent to Bing, and claims were manually extracted from the returned snippets.
  • Relevant scientific papers were retrieved and re-ranked with a T5 relevance model, and evidence sentences were taken from PubMed abstracts.
  • Annotators labeled each evidence-claim pair as SUPPORTS, REFUTES or NEUTRAL.

The result is 14,330 pairs. Baselines built on pretrained language models show that training on real, in-domain claims works much better than training on synthetic or open-domain data. Cross-dataset zero-shot tests show that HealthVER transfers best to other datasets. This gives a harder, more realistic benchmark for verifying health misinformation against science.

Key points

  • Dataset contents: HealthVER has 14,330 evidence-claim pairs with three labels (SUPPORTS, REFUTES, NEUTRAL). Claims average 19 words, and evidence comes from PubMed abstracts.
  • Question sources: The 80 questions came from TREC-COVID, the TREC Health Misinformation Track and Bing "people also ask" suggestions. Claims were mined from the top-40 Bing snippets per question and validated by three biomedical NLP experts.
  • Model results: T5 gave the best results, and in-domain encoders such as SciBERT and BioBERT beat general BERT. Accuracy fell sharply when models saw only the claim without the evidence.
  • Generalization: In pairwise zero-shot transfer, HealthVER had an average accuracy delta of +16.86, or +25.01 after adjusting for training set size. The other datasets (e.g., FEVER, SciFact, PubHealth) ranged from -17.66 to -2.38.
  • Common errors: Mistakes clustered around low lexical overlap, synonym and hypernym mismatches (COVID-19 vs. Coronavirus), unrecognized abbreviations (ACEIs vs. ACE), and claims containing several facts.

Interesting ideas

  • I1. Mining claims from search snippets: Sending common user questions to a search engine and pulling claims from the snippets yields naturally phrased assertions cheaply. The same approach could collect realistic claims in Ukrainian or other low-resource settings without inventing them by hand.
  • I2. Size-adjusted cross-dataset transfer: Every dataset is trained on and tested against every other, and the accuracy deltas are corrected for training set size. This is a fairer way to compare how well datasets generalize than raw in-domain scores.
  • I3. Measuring claim complexity: An ensemble of relation extraction tools estimates how many facts each claim contains. This gives a measurable proxy for difficulty that can be used to compare corpora or to stratify evaluation.
  • I4. Claim-only ablation: Removing the evidence and checking how far performance drops exposes label leakage through surface cues in the claims. This is a cheap sanity check for any NLI-style or verification dataset.
  • I5. Recording annotator disagreement: The authors document cases where reasonable annotators disagree, such as a temperature-related claim that could be labeled support, neutral or refute. Explicitly logging such ambiguous examples helps set realistic ceilings and informs annotation guidelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Institutes of HealthBethesda · United States9 works · 3,833 citations🔬United States National Library of MedicineBethesda · United States5 works · 5,321 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

Cite

Cite as Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet and Dina Demner-Fushman. 2021. Evidence-based Fact-Checking of Health-related Claims. Findings of the Association for Computational Linguistics: EMNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.297

@inproceedings{sarrouti2021evidence,
  title = {{Evidence-based Fact-Checking of Health-related Claims}},
  author = {Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet and Dina Demner-Fushman},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.297},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.297}
}

References

35 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Langu… our card
  2. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Langua… our card
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. our card
  4. Zapan Barua, Sajib Barua, Salma Aktar, Najma Kabir, and Mingze Li. 2020. Effects of misinformation on COVID-19 individual responses and recommendations for resilience of disastrous consequences of misinformation. Progress in Disaster Science, 8:100119. find
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Pretrained language model for scientific text. In EMNLP. find
  6. Jacob Cohen. 1968. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. find
  7. Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, and Preslav Nakov. 2020. Prta: A system to support the analysis of propaganda techniques in the news. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrati… find
  8. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76… our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  10. Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. Generating fact checking briefs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7147–7161, Online. As… find
  11. Roya Ghasemian, Amir Shamshirian, Keyvan Heydari, Mohammad Malekan, Reza Alizadeh-Navaei, Mohammad Ali Ebrahimzadeh, Hamed Jafarpour, Arash Rezaei Shahmirzadi, Mehrdad Khodabandeh, Benyamin Seyfari, Alireza Motamedzadeh, Ehsan Dadgostar, Marzieh Aalinezhad, Meghdad Sedaghat, Nazanin Razzaghi, Bahman… find
  12. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 493–503, Hong Kong, China. Association for… our card
  13. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Wo… our card
  14. Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, Online. Association for Computational Linguistics. our card
  15. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. our card
  16. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60. trast… our card
  17. David O. Meltzer, Thomas J. Best, Hui Zhang, Tamara Vokes, Vineet Arora, and Julian Solway. 2020. Association of vitamin d deficiency and treatment with COVID-19 incidence. find
  18. Rahul Mishra, Dhruv Gupta, and Markus Leippold. 2020. Generating fact checking summaries for web claims. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 81–90, Online. Association for Computational Linguistics. find
  19. Salman Bin Naeem, Rubina Bhatti, and Aqsa Khan. 2020. An exploration of how fake news is taking over social media and putting public health at risk. Health Information & Libraries Journal. find
  20. Preslav Nakov, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. In Lecture Notes in Comput… find
  21. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics. find
  22. Gordon Pennycook, Jonathon McPhetres, Yunhao Zhang, Jackson G. Lu, and David G. Rand. 2020. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7):770–780. find
  23. Lahari Poddar, Wynne Hsu, Mong Li Lee, and Shruti Subramaniyam. 2018. Predicting stances in twitter conversations for detecting veracity of rumors: A neural approach. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (IC-TAI), pages 65–72. find
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. our card
  25. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937, Copenhagen, Denmark. As… our card
  26. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con-Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. arXiv preprint arXiv:1906.03158. our card
  27. James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. find
  28. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359, Santa Fe, New Mexico, USA. Association for Computational Linguistics. our card
  29. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… our card
  30. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22, Baltimore, MD, USA. Association for Computational Linguistics. our card
  31. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association… our card
  32. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wad… our card
  33. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics. our card
  34. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… our card
  35. Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeño, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, Kareem Darwish, James Glass, and Preslav Nakov. 2019. Tanbih: Get to know what you are reading. In Proceedings of the 2019 Conference on… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sarrouti-2021-evidence-based-fact-checking-health/