Elamur Research⌕
☰
home / papers / 2018 / Automated Fact Checking: Task Formulations, Methods and Future Directions

Automated Fact Checking: Task Formulations, Methods and Future Directions

Summary

Work on automatically judging whether a claim is true is spread across NLP, machine learning, knowledge representation, databases and journalism. These communities often name the same problem differently. For example, checking whether a headline matches its article body has been called fake news detection, stance classification and incongruent headline detection. Thorne and Vlachos survey this work and propose a shared framing. From journalism they adopt a split between verification (confirming the source, date and location of material) and fact checking (examining a claim's logic, coherence and context). They also avoid the diluted label "fake news". They then compare approaches along three axes: the input being checked, the kind of verdict returned, and the evidence consulted. The type of evidence turns out to be the main factor that distinguishes methods. The paper ends with a research agenda: open-world retrieval and how to evaluate it, verifying evidence together with claims, handling complex and document-level claims, generating rationales, and using multimodal evidence. It serves as a reference map for anyone designing claim-verification datasets or systems.

Key points

  • Inputs fall into three groups. The first is subject-predicate-object triples checked against knowledge bases such as Freebase. The second is short textual claims, like those curated by PolitiFact and Full Fact. The third is whole documents, where claims must first be extracted.
  • The survey adopts the four-way claim taxonomy from the HeroX fact checking challenge: numerical claims, entity/event properties, position statements and quote verification.
  • Misinformation is treated as the broader category, and disinformation as the subset that adds intent to mislead. Automated fact checking can flag the former but cannot tell the two apart.
  • FEVER is a notable large-scale resource, but its claims are short (about 8 words on average) and its justifications are limited to selected Wikipedia sentences. Human fact checkers, by contrast, synthesize information from many sources.
  • The authors warn that real journalistic checks, such as the "£350 million per week" Brexit claim, need reasoning beyond current systems. They argue this should temper promises made to society.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{thorne2018automated,
  title = {{Automated Fact Checking: Task Formulations, Methods and Future Directions}},
  author = {Thorne, James and Vlachos, Andreas},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  doi = {10.17863/cam.78745},
  url = {https://doi.org/10.17863/cam.78745}
}

References

74 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Technical report, National Bureau of Economic Research. find
  2. Gabor Angeli and Christopher D. Manning. 2014. NaturalLI : Natural Logic Inference for Common Sense Reasoning. Proceedings of EMNLP, pages 534–545. find
  3. Mevan Babakar and Will Moy. 2016. The State of Automated Factchecking. Technical report. find
  4. Hannah Bast, Björn Buchhold, and Elmar Haussmann. 2015. Relevance scores for triples from type-like relations. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 243–252. ACM. find
  5. Hannah Bast, Björn Buchhold, and Elmar Haussmann. 2017. Overview of the Triple Scoring Task at the WSDM Cup 2017. WSDM Cup. find
  6. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. SIGMOD 08 Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. 9 https://fullfact.org/… ↗
  7. S Brin and L Page. 1998. The anatomy of a large scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1/7):107–17. find our card
  8. Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly. 2016. Stop Clickbait: Detecting and preventing clickbaits in online news media. Proceedings of the 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2016, pages 9–16. find
  9. Yimin Chen, Niall J. Conroy, and Victoria L. Rubin. 2015. Misleading Online Content. Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection - WMDD ’15, (November):15–19. find
  10. Sophie Chesney, Maria Liakata, Massimo Poesio, and Matthew Purver. 2017. Incongruent Headlines: Yet Another Way to Mislead Your Readers. In Natural Language Processing meets Journalism workshop at EMNLP 2017, number 1, pages 56–61. find
  11. Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational fact checking from knowledge networks. PLoS ONE, 10(6):1–13. find
  12. Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011. Computational Journalism: a call to arms to database researchers. Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011) Asilomar, California, USA., (January):148–151. find
  13. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  14. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. find
  15. Leon Derczynski and Kalina Bontcheva. 2015. Veracy in Digital Social Networks. Pheme.eu. find
  16. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. find our card
  17. Deutsche Welle. 2017. Germany to force facebook, twitter to delete hate speech. http://www.dw.com/ en/germany-to-force-facebook-twitter-to-delete-hate-speech/a-37927085. Accessed: 2018-09-30. ↗
  18. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation. In Proceedings of the Fourth Conference on Language Resources and Evaluation. find our card
  19. Xin Luna Dong, Evgeniy Gabrilovich, Kevin Murphy, Van Dang, Wilko Horn, Camillo Lugaresi, Shaohua Sun, and Wei Zhang. 2015. Knowledge-based trust: Estimating the trustworthiness of web sources. Proc. VLDB Endow., 8(9):938–949, May. find
  20. Vanessa Wei Feng and Graeme Hirst. 2013. Detecting Deceptive Opinions with Profile Compatibility. Proceedings of the Sixth International Joint Conference on Natural Language Processing, (October):338–346. find
  21. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, (July):171–175. find
  22. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1168, San Diego, California, June. Associati… find our card
  23. Diane Francis and Full Fact. 2016. Fast & furious fact check challenge. https://www.herox.com/ factcheck/. Accessed: 2018-09-30. ↗
  24. Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0), June. find
  25. Swapna Gottipati, Minghui Qiu, Liu Yang, Feida Zhu, and Jing Jiang. 2013. Predicting User ’ s Political Party. pages 177–191. find
  26. Lucas Graves. 2018. Understanding the Promise and Limits of Automated Fact-Checking. Technical report, Reuters Institute, University of Oxford. find
  27. Z Gyöngyi, H Garcia-Molina, and J Pedersen. 2004. Combating web spam with TrustRank. Proceedings of the Thirtieth international conference on Very large data bases, 30:576–587. find
  28. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find our card
  29. Zhen Hai, Peilin Zhao, Peng Cheng, Peng Yang, Xiao-Li Li, and Guangxia Li. 2016. Deceptive Review Spam Detection via Exploiting Task Relatedness and Unlabeled Data. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-16), pages 1817–1826. find
  30. Andreas Hanselowski, Avinesh P.V.S, Benjamin Schiller, and Felix Caspelherr. 2017. Description of the system developed by team athene in the fnc-1. Technical report. find
  31. Naeemul Hassan, Bill Adair, James Hamilton, Chengkai Li, Mark Tremayne, Jun Yang, and Cong Yu. 2015a. The quest to automate fact-checking. In Proceedings of the 2015 Computation+Journalism Symposium. find
  32. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015b. Detecting Check-worthy Factual Claims in Presidential Debates. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM. find
  33. Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017a. Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1803–1812. ACM. find
  34. Naeemul Hassan, Anil Kumar Nayak, Vikas Sable, Chengkai Li, Mark Tremayne, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, and Aaditya Kulkarni. 2017b. ClaimBuster: the first-ever end-to-end fact-checking system. Proceedings of the VLD… find
  35. Xinyu Hua and Lu Wang. 2017. Understanding and detecting supporting arguments of diverse types. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 203–208. Association for Computational Linguistics. find
  36. Ken Hyland. 2015. Metadiscourse. The International Encyclopedia of Language and Social Interaction, (April 2015):1–11. find
  37. Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 1148–1158, Stroudsburg, PA, USA. Association for Compu… find
  38. Garth S. Jowett and Victoria O’ Donnell. 2006. What is propaganda, and how does it differ from persuasion? In Propaganda and Misinformation, chapter 1. Sage Publishers. find
  39. Bill Kovach and Tom Rosenstiel. 2014. The elements of journalism: What newspeople should know and the public should expect. Three Rivers Press (CA). find
  40. N Lao, T Mitchell, and WW Cohen. 2011. Random walk inference and learning in a large scale knowledge base. Proceedings of the Conference . . . , (March):529–539. find
  41. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 158–167. find
  42. Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 1867–1870, New York, NY, USA. ACM. find
  43. Yunfei Long, Qin Lu, Yue Xiao, Minglei Li, and Chu Ren Huang. 2016. Domain-specific user preference prediction based on multiple user activities. Proceedings - 2016 IEEE International Conference on Big Data, Big Data 2016, pages 3913–3921. find
  44. Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake News Detection Through Multi-Perspective Speaker Profiles. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2:252–256. find
  45. Alexios Mantzarlis. 2015. Will verification kill fact-checking? https://www.poynter.org/news/ will-verification-kill-fact-checking. Accessed: 2018-09-30. ↗
  46. Paul McNamee and Hoa Trang Dang. 2009. Overview of the tac 2009 knowledge base population track. find
  47. Rada Mihalcea and Carlo Strapparava. 2009. The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language. Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, (August):309–312. find
  48. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, (August):1003–1011. find
  49. Ndapandula Nakashole and Tom M Mitchell. 2014. Language-Aware Truth Assessment of Fact Candidates. Acl, pages 1009–1019. find
  50. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 309–319, Stroudsburg… find our card
  51. Myle Ott, Claire Cardie, and Jeffrey T. Hancock. 2013. Negative deceptive opinion spam. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 497–501. Association for Computational Linguistics. find
  52. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The Development and Psychometric Properties of LIWC2015. Environment and Planning D: Society and Space. find
  53. Verónica Pérez-Rosas and Rada Mihalcea. (September):1120–1125. 2015. find
  54. Experiments in Open Domain Deception Detection. find
  55. Dean Pomerleau and Delip Rao. 2017. Fake News Challenge. \\url{http://fakenewschallenge.org/}. find
  56. Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait Detection. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9626(1):810–817. find
  57. Vahed Qazvinian, Emily Rosengren, Dragomir R Radev, and Qiaozhu Mei. 2011. Rumor has it : Identifying Misinformation in Microblogs. In Empirical Methods in Natural Language Processing (EMNLP), pages 1589– 1599. find our card
  58. Jacquelyn Rahman. 2012. The n word: Its history and use in the african american community. Journal of English Linguistics, 40(2):137–171. find
  59. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937. Association for Computa… find our card
  60. Benjamin Riedel, Isabelle Augenstein, George Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. CoRR, abs/1707.03264. find
  61. Ivor Shapiro, Colette Brin, Isabelle Bedard-Brule, and Kasia Mychajlowycz. 2013. Verification as a strategic ritual. Journalism Practice, 7(6):657–673. find
  62. Craig Silverman. 2015. Lies , Damn Lies , and Viral Content: How News Websites Spread (and Debunk) Online Rumors, Unverified Claims, and Misinformation. Columbia Journalism School. find
  63. Craig Silverman. 2016. Verification handbook: Additional materials. http://verificationhandbook.com/additionalmaterial/. Accessed: 2018-09-30. ↗
  64. James Thorne and Andreas Vlachos. 2017. An Extensible Framework for Verification of Numerical Claims. In European Chapter of the Association for Computational Linguistics (EACL), pages 37–40. find
  65. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT. find our card
  66. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22, Baltimore, MD, USA, June. Association for Computational Linguistics. find our card
  67. Andreas Vlachos and Sebastian Riedel. 2015. Identification and Verification of Simple Claims about Statistical Properties. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, (September):2596–2601. find
  68. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
  69. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2017. Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering. 1:1–13. find
  70. William Yang Wang. 2017. ”Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. find our card
  71. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT ’05, (October):347–354. find our card
  72. W. A. Woods. 1973. Progress in natural language understanding: An application to lunar geology. In Proceedings of the June 4-8, 1973, National Computer Conference and Exposition, AFIPS ’73, pages 441–450, New York, NY, USA. ACM. find
  73. Lina Zhou, Judee K. Burgoon, Jay F. Nunamaker, and Doug Twitchell. 2004. Automating Linguistics-Based Cues for detecting deception in text-based asynchronous computer-mediated communication. Group Decision and Negotiation, 13(1):81–106. find
  74. Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and resolution of rumours in social media: A survey. ACM Comput. Surv., 51(2):32:1–32:36, February. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thorne-2018-automated-fact-checking-task-formulations/