Elamur Research
☰
home / papers / 2019 / MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1475.pdf

Summary

Automatic fact checking has mostly relied on either small collections of real claims or large but synthetic resources such as FEVER, whose claims are rewritten Wikipedia sentences without real-world context. MultiFC addresses this gap with a large English dataset of genuine claims gathered from fact checking websites. Each claim carries its journalist-assigned verdict, metadata such as speaker, tags and dates, the source article with its outlinks, and up to ten Google Search results to serve as evidence. Every website keeps its own label scheme, so the authors treat each one as a separate domain and frame veracity prediction as multi-task learning. They also propose a model that jointly learns to rank evidence pages and to predict veracity. Encoding evidence and metadata both improve results. The best system reaches a Macro F1 of 49.2%, which leaves considerable room for improvement. The resource matters because it brings realistic, metadata-rich claims to evidence-based verification at scale.

Key points

  • The authors crawled 43,837 claims from English fact checking sites listed by Duke Reporters' Lab and Wikipedia. Cleaning reduced this to 36,534 claims. After dropping rare labels (fewer than 5 occurrences), 34,918 claims from 26 domains remain, split 80/10/10 with label stratification.
  • The number of labels per domain ranges from 2 to 27. Labels are not mapped to a shared scale. Domain names are abbreviated, for example pomt, snes and tron.
  • Evidence was collected by sending each claim verbatim to the Google Search API and keeping the top 10 hits. For each hit, the dataset stores the title, rank, URL, snippet and full page. Entities in claims were obtained via entity linking.
  • The best model, "crawled ranked + meta" trained with MTL and label embeddings, scores 0.625 Micro F1 and 0.492 Macro F1. For comparison, MTL alone reaches 0.448 Macro F1 and STL reaches 0.388. Evidence-based models clearly beat claim-only ones.
  • In the metadata ablation, tags alone gave the largest single gain, and using all metadata together beat using none (0.492 vs 0.441 Macro F1). Domains with many labels and long, complex claims remain the hardest cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

132 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1475

@inproceedings{augenstein2019multifc,
  title = {{MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims}},
  author = {Augenstein, Isabelle and Lioma, Christina and Wang, Dongsheng and Lima, Lucas Chaves and Hansen, Casper and Hansen, Christian and Simonsen, Jakob Grue},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1475},
  url = {https://doi.org/10.18653/v1/d19-1475}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016a. Stance Detection with Bidirectional Conditional Encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876–885, Austin, Texas. Association for Computational Lingui… find our card
  2. Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. 2018. Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces. In NAACL-HLT, pages 1896–1906. Association for Computational Linguistics. find
  3. Isabelle Augenstein, Andreas Vlachos, and Kalina Bontcheva. 2016b. USFD at SemEval-2016 Task 6: Any-Target Stance Detection on Twitter with Autoencoders. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 389–393, San Diego, California. Association for Com… find
  4. Joan Bachenko, Eileen Fitzpatrick, and Michael Schonwetter. 2008. Verification and implementation of language-based deception indicators in civil and criminal narratives. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 41–48. Association for Computati… find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of ICLR. find our card
  6. Ramy Baly, Mitra Mohtarami, James R. Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating Stance Detection and Fact Checking in a Unified Corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  7. Alberto Barrn-Cedeo, Tamer Elsayed, Reem Suwaileh, Llus Mrquez, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 2: Factuality. In CLEF (W… find
  8. Rich Caruana. 1993. Multitask Learning: A Knowledge-Based Source of Inductive Bias. In Proceedings of ICML. find
  9. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims. In Proceedings of NAACL. find
  10. G L Ciampaglia, P Shiralkar, L M Rocha, J Bollen, F Menczer, and A Flammini. 2015. Computational Fact Checking from Knowledge Networks. PLoS One, 10(6). find
  11. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76… find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. find
  13. Omar Enayet and Samhaa R. El-Beltagy. 2017. NileTMRG at SemEval-2017 Task 8: Determining Rumour and Veracity Support for Rumours on Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 470–474. Association for Computational Linguistics. find
  14. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In HLT-NAACL, pages 1163–1168. The Association for Computational Linguistics. find
  15. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In ACL (1), pages 328–339. Association for Computational Linguistics. find
  16. Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koychev. 2017. We Built a Fake News / Click Bait Filter: What Happened Next Will Blow Your Mind! In RANLP 2017, pages 334–343. find
  17. Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017. Turing at SemEval-2017 Task 8: Sequential Approach to Rumour Stance Classification with Branch-LSTM. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 475–480, Vancouver, Canada. Association fo… find
  18. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-End Neural Entity Linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519–529. Association for Computational Linguistics. find
  19. Rada Mihalcea and Carlo Strapparava. 2009. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 309– 312. Association for Computational Linguistics. find
  20. Tsvetomila Mihaylova, Preslav Nakov, Lluı́s Màrquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadzhov, and James R. Glass. 2018. Fact Checking in Community Forums. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 20… find
  21. Tanushree Mitra and Eric Gilbert. 2015. Credbank: A large-scale social media corpus with associated credibility annotations. In ICWSM, pages 258–267. find
  22. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural Language Inference by Tree-Based Convolution and Heuristic Matching. In ACL (2). The Association for Computer Linguistics. find our card
  23. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic Detection of Fake News. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3391–3401. Association for Computational Linguistics. find our card
  24. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL-HLT, pages 2227–2237. Association for Computational Linguistics. find
  25. Dean Pomerleau and Delip Rao. 2017. The Fake News Challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. http://www.fakenewschallenge. org/. Accessed: 2019-02-14. ↗
  26. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016. Credibility Assessment of Textual Claims on the Web. In CIKM, pages 2173–2178. find
  27. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL (2), pages 784–789. Association for Computational Linguistics. find our card
  28. Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. CoRR, abs/1707.03264. find
  29. Victoria Rubin, Niall Conroy, Yimin Chen, and Sarah Cornwell. 2016. Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News. In Proceedings of the Second Workshop on Computational Approaches to Deception Detection, pages 7– 17. Association for Computational Linguistics. find
  30. K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu. 2018. FakeNewsNet: A Data Repository with News Content, Social Context and Dynamic Information for Studying Fake News on Social Media. ArXiv e-prints. find
  31. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
  32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In BlackboxNLP@EMNLP, pages 353–355. Association for Computational Linguistics. find our card
  33. William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. Association for Computational Linguistics. find our card
  34. Wenpeng Yin and Dan Roth. 2018. TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 105–114, Brussels, Belgium. Association for Computational Linguistics. find
  35. Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, and Isabelle Augenstein. 2018. Discourse-aware rumour stance classification in social media using sequential classifiers. Informatino Processing & Management, 54(2):273–290. find
  36. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLOS ONE, 11(3):1–29. find
  37. Eugenio Tacchini, Gabriele Ballarin, Marco L. Della Vedova, Stefano Moret, and Luca de Alfaro. 2017. Some Like it Hoax: Automated Fake News Detection in Social Networks. In Proceedings of the Second Workshop on Data Science for Social Good (So-Good), volume 1960 of CEUR Workshop Proceedings. find
  38. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  39. Andreas Vlachos and Sebastian Riedel. 2014. Fact Checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. Association for Computational Linguistics. find our card