Elamur Research⌕
☰
home / papers / 2020 / Generating Fact Checking Explanations

Generating Fact Checking Explanations

Summary

Most automated fact checking research predicts whether a claim is true. The hardest step for human fact checkers, writing a justification for the verdict, has stayed manual. Atanasova et al. present the first study of producing such veracity explanations automatically. They frame it as extractive summarisation. Given a claim and the long PolitiFact report about it (the ruling comments), a DistilBERT-based model selects the sentences that best approximate the journalists' short justification. The training targets are "oracle" sentences, chosen greedily by ROUGE-2 overlap with the gold justification. A second DistilBERT classifier assigns one of six veracity labels. A joint variant links both tasks through a cross-stitch layer and optimises a weighted sum of the two losses. On LIAR-PLUS, training both objectives together improved veracity prediction over separate training. Human evaluators also rated the jointly learned explanations higher on informativeness, coverage and overall quality. The work matters because it moves fact checking from bare labels toward readable reasoning that does not rely on speaker metadata.

Key points

  • Data: LIAR-PLUS (Alhindi et al., 2018), built from PolitiFact, contains 12,836 statements. After filtering out reports with fewer than three sentences, the split is 10,146 train, 1,278 validation and 1,255 test instances.
  • Input vs. target length: ruling comments average about 39 sentences (904 words), while justifications average about 4 sentences (89 words).
  • Explanation setup: the k = 4 oracle sentences are selected by ROUGE-2 F1. At inference, the model outputs the top n = 4 sentences, scored with a sigmoid layer.
  • Models: Explain-Extr (explanation only), a six-way veracity classifier (true, mostly-true, half-true, barely-true, false, pants-on-fire), and the multi-task model with Explain-MT explanations, which shares DistilBERT representations through a cross-stitch layer.
  • Manual veracity labelling from explanations: annotators agreed with the correct gold label 30.0% of the time with Explain-MT explanations, 23.7% with Explain-Extr, and 40.3% with human justifications. This suggests joint training makes explanations more useful for recovering the right verdict.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.656

@inproceedings{atanasova2020generating,
  title = {{Generating Fact Checking Explanations}},
  author = {Atanasova, Pepa and Simonsen, Jakob Grue and Lioma, Christina and Augenstein, Isabelle},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.656},
  url = {https://doi.org/10.18653/v1/2020.acl-main.656}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity Checks for Saliency Maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pages 9525– 9536, USA. Curran Associates Inc. find
  2. Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 85–90, Brussels, Belgium. Association for Computational Linguistics. find
  3. Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. 2019. Evaluating recurrent neural network explanations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 113–126, Florence, Italy. Association for Computational Linguisti… find
  4. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Langua… find our card
  5. Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677. find
  6. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 103–108, Brussels, Belgi… find our card
  7. Andrew F Hayes and Klaus Krippendorff. 2007. Answering the Call for a Standard Reliability Measure for Coding Data. Communication methods and measures, 1(1):77–89. find
  8. Hamid Karimi, Proteek Roy, Sari Saba-Sadiya, and Jiliang Tang. 2018. Multi-source multi-class fake news detection. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1546–1557, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  9. Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3728–3738, Hong Kong, China. Assoc… find
  10. Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake News Detection Through Multi-Perspective Speaker Profiles. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 252–256. find
  11. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31… find our card
  12. Ilya Loshchilov and Frank Hutter. 2017. Fixing Weight Decay Regularization in Adam. arXiv preprint arXiv:1711.05101. find
  13. Tuhin Chakrabarty, Tariq Alhindi, and Smaranda Muresan. 2018. Robust document retrieval and individual evidence modeling for fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 127– 131, Brussels, Belgium. Association for Computat… find
  14. Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Detect Rumor and Stance Jointly by Neural Multi-task Learning. In Companion Proceedings of the The Web Conference 2018, WWW ’18, pages 585–593, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  16. Christopher Malon. 2018. Team papelo: Transformer networks at FEVER. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 109–113, Brussels, Belgium. Association for Computational Linguistics. find
  17. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogniti… find
  18. Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluı́s Màrquez, and Alessandro Moschitti. 2018. Automatic stance detection using end-to-end memory networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… find our card
  19. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pages 3075–3081. AAAI Press. find our card
  20. Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 340–350, Melbourne, A… find
  21. William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. find our card
  22. Brian Xu, Mitra Mohtarami, and James R. Glass. 2018. Adversarial Domain Adaptation for Stance Detection. In Proceedings of the Thirty-second Annual Conference on Neural Information Processing Systems (NeurIPS)–Continual Learning. find
  23. Fan Yang, Shiva K Pentyala, Sina Mohseni, Mengnan Du, Hao Yuan, Rhema Linder, Eric D Ragan, Shuiwang Ji, and Xia Hu. 2019. XFake: Explainable Fake News Detector with Visualizations. In The World Wide Web Conference, pages 3600–3604. find
  24. Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 97–102, Brussels, Belgium. Association for Co… find
  25. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942, Florence, Italy. Association for Computat… find our card
  26. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent Multi-Task Architecture Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4822–4829. find
  27. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS’2019. find
  28. Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019. dEFEND: Explainable Fake News Detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 395–405. find
  29. Dominik Stammbach and Guenter Neumann. 2019. Team DOMLIN: Exploiting Evidence Enhancement for the FEVER Shared Task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 105–109, Hong Kong, China. Association for Computational Linguistics. find
  30. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/atanasova-2020-generating-fact-checking-explanations/