Elamur Research⌕
☰
home / papers / 2017 / Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

Summary

The paper looks at whether the wording of a text reveals how trustworthy it is. It treats truth as graded rather than as a simple true/false split. The authors built a corpus contrasting trusted English Gigaword articles with articles from seven unreliable sites, grouped as satire, hoax, or propaganda. They compared the groups using stylistic lexicons: LIWC, a subjectivity lexicon, hedge words, and new intensifier lists compiled from Wiktionary.

Fake news uses more first- and second-person pronouns, superlatives, subjective words, modal adverbs, and hedges. Trusted news leans on numbers, money terms, comparatives, and assertive verbs. A MaxEnt classifier on tf-idf n-grams sorts articles into four reliability classes with 0.65 F1 when the test sources are unseen.

A second case study predicts PolitiFact's 6-point ratings for political quotes. This task proves much harder: the best test macro-F1 is about 0.22 for six classes and 0.56 for two. The work matters because it measures style-based reliability cues across several kinds of untrustworthy text, and the authors released all of its resources publicly.

Key points

  • News corpus: trusted articles come from Gigaword. Unreliable articles come from:
    • satire: The Onion, The Borowitz Report, Clickhole
    • hoax: American News, DC Gazette
    • propaganda: The Natural News, Activist Report
  • Statistical testing: the reported feature ratios (fake vs. trusted) are significant at p < 0.01, using a Welch t-test with Bonferroni correction.
    • Swear words (ratio 7.00) and second-person "you" (6.73) are most inflated in fake news.
    • Numbers (0.43), "hear" words (0.50), and money terms (0.57) are most typical of trusted news.
  • 4-way news reliability classification: a MaxEnt classifier with L2 regularization on tf-idf features (up to trigrams) scores 0.91 F1 on in-domain dev data. It drops to 0.65 on out-of-domain test sources, against a random baseline of 0.26.
  • PolitiFact setup: quotes were split 2575/712/1074 for train/dev/test, with all quotes from one speaker kept in the same split.
  • PolitiFact models: an LSTM (GloVe 100-dim initialization, 300-dim hidden state) was compared with MaxEnt and Naive Bayes.
    • Adding LIWC features lifts NB and MaxEnt to roughly the LSTM's level.
    • The same features do not help the LSTM.

Interesting ideas

  • I1. Two-axis taxonomy: unreliable content is classified by author intent (whether they want to deceive) and by factual quality. This separates satire from propaganda and hoaxes, and the same scheme could organize misinformation datasets in other languages.
  • I2. Source-disjoint splits: news articles are split by outlet, and PolitiFact quotes by speaker. This stops models from scoring well by memorizing author style. The large gap between in-domain and out-of-domain results shows how much this choice matters for honest evaluation.
  • I3. Ratio-of-averages lexicon profiling: the authors compare per-document lexicon counts between classes and apply significance tests with multiple-comparison correction. This is a cheap, interpretable way to describe a genre before building classifiers.
  • I4. Crowd-built lexicons: intensifier lists (comparatives, superlatives, action, manner, and modal adverbs) were harvested from Wiktionary. This trick transfers well to lower-resource languages where curated stylistic lexicons are missing.
  • I5. Probing a neural model with lexicon features: concatenating LIWC features to model inputs tests whether a neural model already captures those signals. Here the features gave no gain for the LSTM, suggesting the information was redundant, while the same features gave substantial gains to linear models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

202 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 189 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1317

@inproceedings{rashkin2017truth,
  title = {{Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking}},
  author = {Rashkin, Hannah and Choi, Eunsol and Jang, Jin Yea and Volkova, Svitlana and Choi, Yejin},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1317},
  url = {https://doi.org/10.18653/v1/d17-1317}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media. find
  2. Prakhar Biyani, Kostas Tsioutsiouliklis, and John Blackmer. 2016. ”8 amazing secrets for getting more clicks”: Detecting clickbaits in news streams using article informality. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, pages 94–100. find
  3. David B. Buller and Judee K. Burgoon. 1996. Interpersonal deception theory. Communication Theory 6(3):203–242. https://doi.org/10.1111/j.1468-2885.1996.tb00127.x. — [doi:10.1111/j.1468-2885.1996.tb00127.x](https://doi.org/10.1111/j.1468-2885.1996.tb00127.x) ↗
  4. Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. Hedge detection as a lens on framing in the gmo debates: A position paper. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics. Association for Compu… find
  5. Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational fact checking from knowledge networks. PLOS ONE 10(6):e0128193. https://doi.org/10.1371/journal.pone.0128193. of the 25th International Conference on World Wide We… — [doi:10.1371/journal.pone.0128193](https://doi.org/10.1371/journal.pone.0128193) ↗
  6. Charles G Lord, Lee Ross, and Mark R Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology 37(11):2098–2109. find
  7. Rada Mihalcea and Carlo Strapparava. 2009. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers. Association for Computational Linguistics, pages 309–312. find
  8. Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin 29(5):665–675. find
  9. Brendan Nyhan and Jason Reifler. 2015. The effect of fact-checking on elites: A field experiment on US state legislators. American Journal of Political Science 59(3):628–640. find
  10. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pag… find our card
  11. Kate Connolly, Angelique Chrisafis, Poppy McPherson, Stephanie Kirchgaessner, Benjamin Haas, Dominic Phillips, Elle Hunt, and James W. Pennebaker, Roger J. Booth, Ryan L. Boyd Michael Safi. 2016. Fake news: An insidi-Boyd, and Martha E. Francis. 2015. Linguistic In-ous trend that’s fast becoming a g… find
  12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word Codruta Girlea, Roxana Girju, and Eyal Amir. 2016. representation. In Proceedings of the 2014 Con-Psycholinguistic features for deceptive role detec-ference on Empirical Methods in Natural Language tion… find
  13. Ken Hyland. 2015. Metadiscourse. In The International Encyclopedia of Language and Social Interaction, John Wiley & Sons, Inc., pages 1–11. find
  14. Nitin Jindal and Bing Liu. 2008. Opinion spam and analysis. In Proceedings of the 2008 International Conference on Web Search and Data Mining. ACM, pages 219–230. find
  15. Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the web: Impact, characteristics, and detection of wikipedia hoaxes. In Proceedings Paul Rayson, Andrew Wilson, and Geoffrey Leech. 2001. Grammatical word class variation within the british national corpus sampler. Language and Co… find
  16. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, page… find our card
  17. Victoria L. Rubin, Yimin Chen, and Niall J. Conroy. 2015. Deception detection for news: Three types of fakes. Proceedings of the Association for Information Science and Technology 52(1):1–4. find
  18. Emily Thorson. 2016. Belief echoes: The persistent effects of corrected misinformation. Political Communication 33(3):460–480. find
  19. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. Association for Computational Linguistics, pages 18–22. find our card
  20. William Yang Wang. 2017. “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the Association for Computational Linguistics Short Papers. Association for Computational Linguistics. find our card
  21. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 347–3… find our card
  22. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems. pages 649–657. find our card
  23. Lina Zhou, Judee K. Burgoon, Jay F. Nunamaker, and Doug Twitchell. 2004. Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communications. Group Decision and Negotiation 13(1):81–106. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/