Elamur Research
☰
home / papers / 2017 / ``Liar, Liar Pants on Fire'': A New Benchmark Dataset for Fake News Detection

``Liar, Liar Pants on Fire'': A New Benchmark Dataset for Fake News Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-2067.pdf

Summary

Research on detecting fake news by statistical methods has been held back by the lack of large labeled benchmarks. Earlier public collections had only a few hundred political claims, which is too few for machine learning. This paper introduces LIAR, a public dataset of 12,836 short political statements gathered through the POLITIFACT.COM API. The statements date mainly from 2007–2016, and PolitiFact editors rated each one on a six-level truthfulness scale. Each statement also comes with speaker metadata, its context and subject, and an analysis report with links to sources. The author treats detection as six-way text classification. Baselines are logistic regression, SVMs, Bi-LSTMs and a Kim-style CNN. The paper also proposes a hybrid CNN that encodes metadata with convolution and a bi-directional LSTM, then combines it with the text representation. Of the text-only models, the CNN scores best. Adding all metadata raises test accuracy further. LIAR is roughly ten times larger than earlier comparable resources and draws on real-world sources rather than crowdsourcing. It can also support fact-checking, stance and rumor research.

Key points

  • Dataset: LIAR contains 12,836 statements, split into 10,269 train, 1,284 validation and 1,283 test. The average statement is 17.9 tokens long. It is downloadable from the author's UCSB page.
  • Labels: the six ratings are pants-fire, false, barely-true, half-true, mostly-true and true. The classes are fairly balanced: pants-fire has 1,050 cases and each other label has 2,063–2,638. On a sample, agreement with the PolitiFact analyses reached Cohen's kappa 0.82.
  • Metadata: each statement includes subject, speaker, job, home state, party, venue and a "credit history" vector. That vector counts the speaker's past ratings and must exclude the current label to avoid leakage.
  • Text-only test accuracy: majority baseline 0.208, LR 0.247, SVM 0.255, Bi-LSTM 0.233 (it overfits), CNN 0.270. The CNN's advantage over the SVM is significant (p < .0001).
  • Hybrid model: the hybrid CNN using text plus all metadata reaches about 0.274 test accuracy, the best result in the paper. The extracted table was garbled, so check this figure and the per-feature results against the original paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

285 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 268 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2067

@inproceedings{wang2017liar,
  title = {{``Liar, Liar Pants on Fire'': A New Benchmark Dataset for Fake News Detection}},
  author = {Wang, William Yang},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2067},
  url = {https://doi.org/10.18653/v1/p17-2067}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Verónica Pérez-Rosas and Rada Mihalcea. 2015. Experiments in open domain deception detection. In EMNLP. pages 1120–1125. find
  2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493–2537. find our card
  3. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. Proceedings of the ACL 2014 Workshop on Language Technology and Computational Social Science . find our card
  4. Koby Crammer and Yoram Singer. 2001. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research 2(Dec):265–292. find
  5. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using \#petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (E… find
  6. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, pages 171–175. find
  7. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems. pages 649–657. find our card
  8. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL. find
  9. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5):602–610. find
  10. Zhen Hai, Peilin Zhao, Peng Cheng, Peng Yang, Xiao-Li Li, Guangxia Li, and Ant Financial. 2016. Deceptive review spam detection via exploiting task relatedness and unlabeled data. In EMNLP. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find