Elamur Research
☰
home / papers / 2014 / Fact Checking: Task definition and dataset construction

Fact Checking: Task definition and dataset construction

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W14-2508.pdf

Summary

The paper frames automated fact checking, which means judging how truthful a claim is, as a formal NLP task. It is motivated by the manual effort journalists spend verifying politicians' statements and by how quickly unverified content spreads on social media. The authors define the task as ordinal classification rather than a simple true/false decision. The speaker and the time of a claim count as part of its context. The task is limited to claims that can be checked objectively.

They built a small public dataset from Channel 4's fact-checking blog and PolitiFact's Truth-O-Meter, mapping both sites' verdicts onto a shared five-level scale. They then discuss simple baselines: supervised classifiers and nearest-neighbour matching against claims already checked. They also outline an interpretable pipeline that mirrors how journalists work. The main contribution is conceptual. It is an early benchmark definition linking fact checking to textual entailment, semantic parsing, information extraction and machine comprehension. Unlike those tasks, the evidence must be located by the system rather than supplied.

Key points

  • Of 221 journalist-checked statements reviewed, 106 were judged suitable. Each record holds the claim, date, speaker, verdict label and URL, plus the sources the journalists consulted.

  • The labels are TRUE, MOSTLY TRUE, HALF TRUE, MOSTLY FALSE and FALSE, unified across the Channel 4 and PolitiFact rating schemes.

  • Several kinds of claims were excluded: causal claims, predictions about the future, non-factual claims, and claims whose verdicts depended on offline evidence such as personal communications.

  • The journalists' written analyses were deliberately left out, because their wording reveals the verdict and would reduce part of the task to sentiment analysis.

  • The proposed four-stage architecture has these steps:

    1. Extract checkable statements.
    2. Formulate questions about them.
    3. Answer those questions from databases or text.
    4. Combine the answers into a verdict.

    The authors highlight interpretability, which they connect to algorithmic accountability, as a key requirement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-2508

@inproceedings{vlachos2014fact,
  title = {{Fact Checking: Task definition and dataset construction}},
  author = {Vlachos, Andreas and Riedel, Sebastian},
  booktitle = {Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science},
  year = {2014},
  doi = {10.3115/v1/w14-2508},
  url = {https://doi.org/10.3115/v1/w14-2508}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *sem 2013 shared task: Semantic textual similarity. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics and the Shared Task: Semantic Textual Similarity, pages 32–43, Atlanta, GA. find our card
  2. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of the 2005 Conference on Empirical Methods in Natural Language Processing (EMNLP 2005), pages 628–635. find our card
  3. Peter Clark, Philip Harrison, and Niranjan Balasubramanian. 2013. A study of the knowledge base requirements for passing an elementary science test. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, pages 37–42. find
  4. Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011. Computational journalism: A call to arms to database researchers. In Proceedings of the Conference on Innovative Data Systems Research, volume 2011, pages 148–151. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, pages 17… find
  6. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301–333, June. find our card
  7. Nick Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. Technical report, Tow Center for Digital Journalism. find
  8. Richard Farkas, Veronika Vincze, Gyorgy Mora, Janos Csirik, and Gyorgy Szarvas. 2010. find
  9. The CoNLL 2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text. In Proceedings of the CoNLL 2010 Shared Task. find our card
  10. Terry Flew, Anna Daniel, and Christina L. Spurgeon. 2012. The promise of computational journalism. In Proceedings of the Australian and New Zealand Communication Association Conference, pages 1– 19. find
  11. Eibe Frank and Mark Hall. 2001. A simple approach to ordinal classification. In Proceedings of the 12th European Conference on Machine Learning, pages 145–156. find
  12. Luke Goode. 2009. Social news, citizen journalism and democracy. New Media & Society, 11(8):1287– 1305. find
  13. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1545–1556, Seattle, WA. find
  14. Sasa Petrovic. 2013. Real-time event detection in massive streams. Ph.D. thesis, School of Informatics, University of Edinburgh. find
  15. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, WA. find our card
  16. Sebastian Riedel, Limin Yao, Benjamin M. Marlin, and Andrew McCallum. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta,… find