Elamur Research⌕
☰
home / papers / 2019 / A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

Paper · 2019 · Andreas Hanselowski, Christian Stab, Claudia Schulz et al.

Summary

Training machine learning systems for automated fact-checking requires large, reliably annotated data covering the whole verification pipeline. The authors argue that earlier resources fall short: some contain only a few hundred claims, some lack evidence annotations, and FEVER, though large, relies on synthetic claims built from Wikipedia. They introduce a new mixed-domain corpus derived from the Snopes fact-checking website. It combines material gathered by Snopes fact-checkers with crowd-sourced annotations of evidence at two granularity levels, evidence sources, and evidence stance. The corpus can be used for each of four pipeline stages: retrieving documents, extracting evidence, detecting stance, and validating claims. The authors document their collection and annotation procedure and report substantial inter-annotator agreement. They then benchmark strong systems from the FEVER shared task and the Fake News Challenge, along with BERT-based models. All systems score noticeably lower than on comparable benchmarks. The error analysis points to heterogeneous web data, evidence from unreliable sources, and a skew toward false claims. The resource therefore offers a more realistic and demanding testbed for fact-checking research.

Key points

  • The corpus contains 6,422 validated natural claims and roughly 14.5k associated documents, with about 14.3k evidence annotations. The material spans blogs, news, and social media.
  • Evidence is annotated at two levels: evidence text snippets (ETSs) and fine-grained evidence (FGE). Annotations also include stance toward the claim and the source of each piece of evidence.
  • For claim validation, Snopes verdicts were mapped onto the three FEVER classes (refuted, supported, NEI). Ambiguous verdicts such as legend or miscaptioned were dropped.
  • BertEmb performed best on claim validation with a macro F1 of 0.485. Systems reach about 0.7 on FEVER. Transfer learning between FEVER and this corpus, in either direction, gave no significant improvement.
  • A crawler that rebuilds the dataset (UKPLab/conll2019-snopes-crawling) and the experiment code are released on GitHub. The original corpus is available on request for non-commercial research via TU Darmstadt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany76 works · 35,008 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

Cite

Cite as Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li and Iryna Gurevych. 2019. A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k19-1046

@inproceedings{hanselowski2019richly,
  title = {{A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking}},
  author = {Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li and Iryna Gurevych},
  booktitle = {Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)},
  year = {2019},
  doi = {10.18653/v1/k19-1046},
  url = {https://doi.org/10.18653/v1/k19-1046}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Darina Benikova, Margot Mieskes, Christian M. Meyer, and Iryna Gurevych. 2016. Bridging the gap between extractive and abstractive summaries: Creation and evaluation of coherent extracts from heterogeneous sources. In Proceedings of the 26th International Conference on Computational Linguistics (COL… find
  3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  4. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M Meyer, and Iryna Gurevych. 2018a. A retrospective analysis of the fake news challenge stance detection task. Proceedings of the 2018 International Committee on Computational Linguistics. our card
  5. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018b. Ukp-athene: Multi-sentence textual entailment for claim verification. Proceedings of the EMNLP 2018 First Workshop on Fact Extraction and Verification. our card
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  7. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations). our card
  8. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), pages 1120–1130, Atlanta, GA, US… our card
  9. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. find
  10. Lee Howell et al. 2013. Digital wildfires in a hyper-connected world. WEF Report, 3:15–94. find
  11. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017). our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NA… our card
  13. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), pages 1163–1168, San Diego, CA, USA. our card
  14. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2017. The argument reasoning comprehension task. Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018). find
  15. Andreas Hanselowski and Iryna Gurevych. 2017. A framework for automated fact-checking for real-time validation of emerging claims on the web. Proceedings of the NIPS Workshop on Prioritising Online Content (WPOC2017). find
  16. Preslav Nakov, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluı́s Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. In Proceedings of the… find
  17. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. our card
  18. Dean Pomerleau and Delip Rao. 2017. The Fake News Challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. http: Ac-//www.fakenewschallenge.org/. cessed: 2019-4-20. find
  19. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 1003–1012. International World Wide… find
  20. Christopher Tauchmann, Thomas Arnold, Andreas Hanselowski, Christian M Meyer, and Margot Mieskes. 2018. Beyond generic summarization: A . multi-faceted hierarchical summarization corpus of large heterogeneous data. In Proceedings of the Eleventh International Conference on Language Resources and Eva… find
  21. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT. our card
  22. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (fever) shared task. arXiv preprint arXiv:1811.10971. our card
  23. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. our card
  24. William Yang Wang. 2017. ” liar, liar pants on fire”: A new benchmark dataset for fake news detection. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). our card
  25. Klaus Zechner. 2002. Automatic Summarization of Open-Domain Multiparty Dialogues in Diverse Genres. Computational Linguistics, 28(4):447–485. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hanselowski-2019-richly-annotated-corpus-different-tasks/