Elamur Research
☰
home / papers / 2018 / Automatic Detection of Fake News

Automatic Detection of Fake News

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/C18-1287.pdf

Summary

Misinformation spreads widely online, and earlier fake news research relied either on satire sources, which carry humor confounds, or on fact-checking sites that cover mostly politics. Pérez-Rosas et al. address this by building two new English datasets across seven domains.

  • FakeNewsAMT pairs 240 manually fact-checked excerpts from mainstream US outlets in six topics with fabricated counterparts. Amazon Mechanical Turk workers wrote the fakes and were asked to imitate journalistic style.
  • Celebrity holds 250 genuine and 250 fake celebrity stories collected directly from the web.

The authors study linguistic contrasts between the classes and train detectors on lexical, syntactic and semantic features. LIWC-based features perform consistently well across domains. The best models reach up to 76% accuracy. Compared with two human annotators, the system does better on the multi-domain crowdsourced data but slightly trails people on celebrity gossip.

The work matters because it releases public, multi-domain resources made of full news excerpts rather than short claims. It also shows that stylistic cues alone clearly beat chance. The authors recommend adding metadata, other modalities and fact verification.

Key points

  • Datasets: FakeNewsAMT has 240 fake and 240 legitimate items, with 40 per domain in sports, business, entertainment, politics, technology and education. Celebrity has 250 fake and 250 legitimate items.
  • Accuracy: the top classifier reaches about 76%.
  • Human baseline:
    • On FakeNewsAMT, annotators scored 0.71 and 0.70, while the system scored 0.74.
    • On Celebrity, annotators scored 0.80 and 0.77, while the system scored 0.76.
    • Inter-annotator Kappa was moderate.
  • LIWC findings:
    • On FakeNewsAMT, legitimate texts use more cognitive-process words, function words, negations and relativity terms. Fake texts lean toward social and positive words, certainty, a present/future focus, and more verbs, adverbs and punctuation.
    • On Celebrity, legitimate texts use more first-person pronouns, time and past-focus terms, and positive emotion words. Fake texts use more he/she pronouns and negative emotion words, with a focus on the present.
  • Availability: both datasets can be downloaded at lit.eecs.umich.edu/downloads.html.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{perezrosas2018automatic,
  title = {{Automatic Detection of Fake News}},
  author = {Pérez-Rosas, Verónica and Kleinberg, Bennett and Lefevre, Alexandra and Mihalcea, Rada},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1287/}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gary D Bond and Adrienne Y Lee. 2005. Language of lies in prison: Linguistic classification of prisoners’ truthful and deceptive natural language. Applied Cognitive Psychology, 19(3):313–329. find
  2. Yimin Chen, Niall J Conroy, and Victoria L Rubin. 2015. News in an online world: The need for an ”automatic crap detector”. Proceedings of the Association for Information Science and Technology, 52(1):1–4. find
  3. Niall J Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the Association for Information Science and Technology, 52(1):1–4. find
  4. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 171–175. Association for Computational Linguistics. find
  5. Eileen Fitzpatrick, Joan Bachenko, and Tommaso Fornaciari. 2015. Automatic detection of verbal deception. Synthesis Lectures on Human Language Technologies, 8(3):1–119. find
  6. Aniko Hannak, Drew Margolin, Brian Keegan, and Ingmar Weber. 2014. Get Back! You Don’t Know Me Like That: The Social Mediation of Fact Checking Interventions in Twitter Conversations. In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media (ICWSM’14), Ann Arbor, MI, June. find
  7. Molly E Ireland and James W Pennebaker. 2010. Language style matching in writing: synchrony in essays, correspondence, and poetry. Journal of personality and social psychology, 99(3):549. find
  8. Gottfried Jeffrey and Shearer Elisa. 2016. News use across social media platforms 2016. In Pew Research Center Reports. find
  9. Zhiwei Jin, Juan Cao, Yu-Gang Jiang, and Yongdong Zhang. 2014. News credibility evaluation on microblog with a hierarchical propagation model. In Data Mining (ICDM), 2014 IEEE International Conference on, pages 230–239. IEEE. find
  10. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 423–430, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  11. Max Kuhn, Jed Wing, Steve Weston, Andre Williams, Chris Keefer, Allan Engelhardt, Tony Cooper, Zachary Mayer, Brenton Kenkel, the R Core Team, Michael Benesty, Reynald Lescarbeau, Andrew Ziem, Luca Scrucca, Yuan Tang, and Can Candan., 2016. caret: Classification and Regression Training. R package ve… find
  12. David Meyer, Evgenia Dimitriadou, Kurt Hornik, Andreas Weingessel, and Friedrich Leisch, 2015. e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.6-7. find
  13. Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 531–538, Stroudsburg, PA, USA. Association for Computat… find
  14. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey Hancock. 2011a. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 309–319, Stroudsburg,… find
  15. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011b. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 309–319, Portland, Oregon, USA, June. A… find
  16. Myle Ott, Claire Cardie, and Jeffrey T Hancock. 2013. Negative deceptive opinion spam. In HLT-NAACL, pages 497–501. find
  17. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015. Technical report. find
  18. Verónica Pérez-Rosas and Rada Mihalcea. 2014. Cross-cultural deception detection. In ACL (2), pages 440–445. find
  19. Verónica Pérez-Rosas and Rada Mihalcea. 2015. Experiments in open domain deception detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1120–1125, Lisbon, Portugal, September. Association for Computational Linguistics. find
  20. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, pages 2173–2178, New York, NY, USA. ACM. find
  21. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2017. A stylometric inquiry into hyperpartisan and fake news. CoRR, abs/1702.05638. find our card
  22. Victoria L. Rubin, Yimin Chen, and Niall J. Conroy. 2015. Deception detection for news: Three types of fakes. Proceedings of the Association for Information Science and Technology, 52(1):1–4. find
  23. Victoria L Rubin, Niall J Conroy, Yimin Chen, and Sarah Cornwell. 2016. Fake news or truth? using satirical cues to detect potentially misleading news. In Proceedings of NAACL-HLT, pages 7–17. find
  24. Wafa Shafqat, Seunghun Lee, Sehrish Malik, and Hyun-chul Kim. 2016. The language of deceivers: Linguistic features of crowdfunding scams. In Proceedings of the 25th International Conference Companion on World Wide Web, pages 99–100. International World Wide Web Conferences Steering Committee. find
  25. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355. find our card
  26. C. Toma and J. Hancock. 2010. Reading between the lines: linguistic cues to deception in online dating profiles. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, CSCW ’10, pages 5–8, New York, NY, USA. ACM. find
  27. Byron C Wallace. 2015. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
  28. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426. Association for Computational Linguistics. find our card
  29. D. Warkentin, M. Woodworth, J. Hancock, and N. Cormier. 2010. Warrants and deception in computer mediated communication. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, pages 9–12. ACM. find
  30. Linfeng Zhang and Yong Guan. 2008. Detecting click fraud in pay-per-click streams of online advertising networks. In Distributed Computing Systems, 2008. ICDCS’08. The 28th International Conference on, pages 77–84. IEEE. find