Elamur Research⌕
☰
home / papers / 2020 / That is a Known Lie: Detecting Previously Fact-Checked Claims

That is a Known Lie: Detecting Previously Fact-Checked Claims

Paper · 2020 · Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino et al.

Summary

The paper argues that a key step in fact-checking has received little research attention: given a check-worthy statement, decide whether a trusted organization has already verified it. Viral claims tend to resurface and politicians repeat themselves, so finding an earlier verdict can save hours of manual work. The authors frame this as a ranking problem: order a pool of verified claims so that those useful for checking the input, or part of it, come first. They release two datasets with code. In one, sentences from political debates and speeches are linked to PolitiFact articles. In the other, tweets are linked to Snopes articles. Unlike earlier resources built on manually normalized claims, the inputs are raw, naturally occurring text. They compare BM25 and sentence-BERT matching against the verified claim, the article title and the article body. A pairwise RankSVM then merges these signals to re-rank the top BM25 candidates. The re-ranker clearly beats the strongest single models on both datasets. This matters because such lookups are fast and could support journalists in real time.

Key points

  • The task is placed as the second step of a verification pipeline:
    • check-worthiness detection,
    • lookup of previously fact-checked claims,
    • evidence retrieval,
    • factuality assessment.
  • Two datasets pair natural input claims with fact-checking articles: PolitiFact (2016 US presidential debates and speeches) and Snopes (tweets). Data and code are at github.com/sshaar/That-is-a-Known-Lie.
  • Sentence-BERT worked much better than plain BERT and also beat the Universal Sentence Encoder. On full articles, the best setup used scores for the top-4 matching body sentences plus the VerClaim and title scores, but it still trailed BM25 slightly.
  • On PolitiFact, the RankSVM re-ranker gained 0.04–0.05 absolute across measures, with the best list depth at top-100. On Snopes, it gained 0.09–0.18, with the best depth at top-50. The first-stage MRR was 0.565 for PolitiFact (IR:Body) and 0.664 for Snopes (IR:VerClaim+Title).
  • On Snopes, matching against the article body was dropped. The input tweets appear verbatim in the articles, and results stayed unrealistically high even after filtering the tweet text out.

Interesting ideas

  • I1. Scores plus ranks as features. Feeding a re-ranker both the raw scores and the reciprocal ranks from several retrievers mixes exact lexical matching with semantic similarity. This cheap trick could transfer to any hybrid BM25 + embedding search system.
  • I2. Matching against several fields. The input was compared separately with the short normalized claim, the title and the best body sentences, rather than with one document vector. This suits long documents whose key point is stated in only a few places.
  • I3. Re-ranking depth tied to first-stage quality. A stronger first-stage retriever needed a shallower candidate list, and going deeper than needed hurt results. List depth should therefore be tuned per dataset rather than fixed.
  • I4. Leakage awareness. When the query text is copied verbatim inside the target documents, simply filtering it out may not be enough to make evaluation honest. Dropping that setting entirely is the safer choice, which is a useful habit when building retrieval benchmarks from scraped articles.
  • I5. Raw inputs for evaluation. Building test sets from claims as people actually phrase them, rather than editor-normalized versions, gives a more realistic picture of system performance. This applies to any claim-matching or FAQ-retrieval resource, including for Ukrainian-language sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria10 works · 2,341 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

Cite

Cite as Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino and Preslav Nakov. 2020. That is a Known Lie: Detecting Previously Fact-Checked Claims. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.332

@inproceedings{shaar2020that,
  title = {{That is a Known Lie: Detecting Previously Fact-Checked Claims}},
  author = {Shaden Shaar, Nikolay Babulkov, Giovanni {Da San Martino} and Preslav Nakov},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.332},
  url = {https://doi.org/10.18653/v1/2020.acl-main.332}
}

References

54 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019a. Applying BERT to document retrieval with Birch. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,… find
  2. Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019b. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P… find
  3. Pepa Atanasova, Lluı́s Màrquez, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims, Task 1: Check-worthine… find
  4. Pepa Atanasova, Preslav Nakov, Georgi Karadzhov, Mitra Mohtarami, and Giovanni Da San Martino. 2019. Overview of the CLEF-2019 CheckThat! Lab on Automatic Identification and Verification of Claims. Task 1: Check-Worthiness. In Working Notes of the Conference and Labs of the Evaluation Forum, CLEF ’1… find
  5. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Langua… our card
  6. Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018a. Predicting factuality of reporting and bias of news media sources. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 3528–3539, Brussels, Belgium. our card
  7. Ramy Baly, Mitra Mohtarami, James Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018b. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… our card
  8. Alberto Barrón-Cedeño, Tamer Elsayed, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, and Preslav Nakov. 2020. Check-That! at CLEF 2020: Enabling the automatic identification and verification of claims on social media. In Proceedings of the European Conference on Informatio… find
  9. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference, TAC ’09, Gaithersburg, MD, USA. find
  10. Kevin R. Canini, Bongwon Suh, and Peter L. Pirolli. 2011. Finding credible information sources in social networks based on content and social structure. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust, and the IEEE International Conference on Social Computin… find
  11. Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web, WWW ’11, pages 675–684, Hyderabad, India. find
  12. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, pages 1–14, Vancouver, Canada. our card
  13. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… our card
  14. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. our card
  15. Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational fact checking from knowledge networks. PLOS ONE, 10(6):1–13. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  17. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP ’05, Jeju Island, Korea. our card
  18. Sebastian Dungs, Ahmet Aker, Norbert Fuhr, and Kalina Bontcheva. 2018. Can rumour stance alone predict veracity? In Proceedings of the 27th International Conference on Computational Linguistics, COLING ’18, pages 3360–3370, Santa Fe, NM, USA. find
  19. Tamer Elsayed, Preslav Nakov, Alberto Barrón-Cedeño, Maram Hasanain, Reem Suwaileh, Pepa Atanasova, and Giovanni Da San Martino. 2019. CheckThat! at CLEF 2019: Automatic identification and verification of claims. In Proceedings of the 41st European Conference on Information Retrieval, ECIR ’19, pa… find
  20. Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019a. ExFaKT: A framework for explaining facts over knowledge graphs and text. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM ’19, pages 87–95, Melbourne, Australia. find
  21. Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019b. Tracy: Tracing facts over knowledge graphs and text. In The World Wide Web Conference, WWW ’19, pages 3516–3520, San Francisco, CA, USA. find
  22. Pepa Gencheva, Preslav Nakov, Lluı́s Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP ’17, pages 267–27… find
  23. Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’19, pag… our card
  24. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM ’15, pages 1835– 1838, Melbourne, Australia. find
  25. Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, Vikas Sable, Chengkai Li, and Mark Tremayne. 2017. ClaimBuster: The first-ever end-to-end fact-checking system. Proc. VLDB Endow., 10(1… find
  26. Viet-Phi Huynh and Paolo Papotti. 2019. A benchmark for fact checking algorithms built on knowledge bases. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, pages 689–698, Beijing, China. find
  27. Georgi Karadzhov, Preslav Nakov, Lluı́s Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. Fully automated fact checking using external sources. In Proceedings of the Conference on Recent Advances in Natural Language Processing, RANLP ’17, pages 344–353, Varna, Bulgaria. find
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. find
  29. Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, IJCAI ’16, pages 3818–3824, New York, NY, U… find
  30. Jing Ma, Wei Gao, and Kam-Fai Wong. 2019. Detect rumors on Twitter by promoting information campaigns with generative adversarial learning. In Proceedings of the World Wide Web Conference, WWW ’19, pages 3049–3055, San Francisco, CA, USA. find
  31. Tsvetomila Mihaylova, Preslav Nakov, Lluı́s Màrquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadjov, and James Glass. 2018. Fact checking in community forums. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI ’18, pages 879–886, New Orleans, LA, USA. find
  32. Sebastião Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchel, and Zita Marinho. 2019. Automated fact checking in the news room. In Proceedings of the World Wide Web Conference, WWW ’19, pages 3579–3583, San Francisco, CA, USA. find
  33. Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analysis in news communities. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 353–362, Melbourne, Australia. find
  34. Moin Nadeem, Wei Fang, Brian Xu, Mitra Mohtarami, and James Glass. 2019. FAKTA: An automatic end-to-end fact checking system. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’19, pages 78–83, Minneapolis, MN, USA. find
  35. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of 2019. ClaimsKG: A knowledge graph of fact-checked claims. In Proceedings of the 18th International Semantic Web Conference, ISWC ’19, pages 309–324, Au… find
  36. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv:1901.04085. our card
  37. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv:1910.14424. — [arXiv:1910.14424](https://arxiv.org/abs/1910.14424) find
  38. Ayush Patwari, Dan Goldwasser, and Saurabh Bagchi. 2017. TATHYA: A multi-classifier system for detecting check-worthy statements in political debates. In Proceedings of the 2017 ACM Conference on Information and Knowledge Management, CIKM ’17, pages 2259–2262, Singapore. find
  39. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… our card
  40. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW ’17 Companion, pages 1003–1012, Perth… find
  41. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP ’17, pages 2931–2937, Copenhagen,… our card
  42. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 3980–399… our card
  43. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  44. Prashant Shiadralkar, Alessandro Flammini, Filippo Menczer, and Giovanni Luca Ciampaglia. 2017. Finding streams in knowledge graphs to support fact checking. In Proceedings of the 2017 IEEE International Conference on Data Mining, ICDM ’17, pages 859–864, New Orleans, LA, USA. find
  45. Andon Tchechmedjiev, Pavlos Fafalios, Katarina Boland, Malo Gasquet, Matthäus Zloch, Benjamin Zapilko, Stefan Dietze, and Konstantin Todorov. find
  46. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NA… our card
  47. Slavena Vasileva, Pepa Atanasova, Lluı́s Màrquez, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. It takes nine to smell a rat: Neural multi-task learning for check-worthiness prediction. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP ’19,… find
  48. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. find
  49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… our card
  50. William Yang Wang. 2017. “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 422–426, Vancouver, Canada. our card
  51. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’18, pa… find
  52. Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Simple applications of BERT for ad hoc document retrieval. arXiv:1903.10972. find
  53. Tauhid Zaman, Emily B. Fox, and Eric T. Bradlow. 2014. A Bayesian approach for predicting the popularity of tweets. Ann. Appl. Stat., 8(3):1583–1611. find
  54. Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-checking meets fauxtography: Verifying claims about images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP ’19, pages 2099–2108, Hong Kong, China. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaar-2020-that-is-known-lie-detecting/