Where is Your Evidence: Improving Fact-checking by Justification Modeling
Paper · 2018 · Tariq Alhindi, Savvas Petridis and Smaranda Muresan
Summary
Automatic fact-checking on the LIAR dataset (12,836 PolitiFact claims, six truth labels) has mostly relied on the claim text and speaker metadata. That ignores the evidence journalists actually used to reach a verdict. The authors build LIAR-PLUS, which adds to each claim a justification taken automatically from its PolitiFact article. They collect the sentences of the concluding summary section or, if there is none, the article's final five sentences, and they remove the sentence that states the verdict.
They compare four input settings: statement only (S), statement with justification (SJ), enriched statement plus metadata (S+M), and everything combined (S+MJ). Each setting is tested with feature-based models (LR, SVM) and neural ones (BiLSTM, and a parallel two-encoder P-BiLSTM). Adding the justification helps every model in both the binary and the six-way task. For example, six-way test F1 for LR rises from 0.25 to 0.37. The result shows that human-written evidence summaries are a strong signal for predicting graded veracity. The resource is public.
Key points
- LIAR-PLUS extends LIAR (Wang, 2017) with justifications extracted automatically from PolitiFact articles.
- It covers the six labels: pants-fire, false, mostly-false, half-true, mostly-true, true.
- It is released at github.com/Tariq60/LIAR-PLUS.
- Features and lexicons:
- For statements, plain unigrams beat tf-idf and GloVe in the feature-based models.
- Enriched features came from SentiStrength, NRC EmoLex and LIWC, plus speaker label-history metadata.
- Six-way results:
- On test, BiLSTM goes from 0.23 to 0.35 F1 and LR from 0.25 to 0.37 F1 when the justification is added.
- On validation, LR in S+MJ reaches 0.38 F1, against 0.26 for S+M.
- Wang's earlier best was about 0.27 F1.
- Binary results (pants-fire, false and mostly-false grouped as FALSE; the other three as TRUE):
- Scores climb from roughly 0.56–0.61 F1 to roughly 0.66–0.71 F1.
- The best test scores are 0.70 (P-BiLSTM, S+MJ) and 0.68 (several models in SJ and S+MJ).
- Per-class effects:
- Without justification, pants-fire is the weakest class.
- With justification, scores become more even across classes, and pants-fire gains the most.
- P-BiLSTM beats a single BiLSTM only in the six-way task.
Interesting ideas
- I1. Mining structured article endings for cheap supervision. Professional articles often end with a recognizable wrap-up section. Pulling that section, with a fixed number of final sentences as a fallback, yields evidence text without manual annotation. This could work for any news or review corpus.
- I2. Removing the verdict sentence to avoid label leakage. Dropping the sentence that states the verdict forces the model to rely on the reasoning itself. The same care applies whenever auxiliary text is built from documents that already contain the answer.
- I3. Separate encoders for paired inputs. In P-BiLSTM, one encoder reads the claim and another reads the evidence, and their outputs are joined before classification. This is a simple baseline for any claim–context task, such as entailment, stance or replies in a dialogue.
- I4. Ablation grid across model families. Testing the same input settings on both linear and neural models shows the gain comes from the data, not from one architecture. This is a good template for judging any new information source.
- I5. Error analysis that tracks one example across input settings. Following the same claim through every input combination shows which signal fixes which mistake. In the paper's examples, concessive markers like "however" and "but" in the evidence point to partially true claims.
Institutions
United States📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Sociology and Political Science · 221 works
Cite
Cite as Tariq Alhindi, Savvas Petridis and Smaranda Muresan. 2018. Where is Your Evidence: Improving Fact-checking by Justification Modeling. Proceedings of the First Workshop on Fact Extraction and VERification (FEVER).
@inproceedings{alhindi2018where,
title = {{Where is Your Evidence: Improving Fact-checking by Justification Modeling}},
author = {Tariq Alhindi, Savvas Petridis and Smaranda Muresan},
booktitle = {Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)},
year = {2018},
doi = {10.18653/v1/w18-5513},
url = {https://doi.org/10.18653/v1/w18-5513}
}References
- Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. find
- Adam: CoRR, Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake news detection through multi-perspective speaker profiles. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers). find
- Saif M Mohammad and Peter D Turney. 2010. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, pages 26–34. Association for Computation… our card
- James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. our card
- Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 231–240. Association for Comput… our card
- Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937. our card
- Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450. our card
- Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the Association for Information Science and Technology, 61(12):2544–2558. find
- J Thorne and A Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018). our card
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT. our card
- Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. our card
- William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Vancouver, BC, Canada. ACL. our card
- Xuezhi Wang, Cong Yu, Simon Baumgartner, and Flip Korn. 2018. Relevant document discovery for fact-checking articles. In Companion of the The Web Conference 2018 on The Web Conference 2018, pages 525–533. International World Wide Web Conferences Steering Committee. find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/alhindi-2018-where-is-your-evidence-improving/