Elamur Research⌕
☰
home / papers / 2018 / A Retrospective Analysis of the Fake News Challenge Stance-Detection Task

A Retrospective Analysis of the Fake News Challenge Stance-Detection Task

Summary

The FNC-1 shared task (2017) asked systems to decide whether a news article agrees with, disagrees with, discusses, or is unrelated to a given headline. Despite 50 participating teams, no follow-up study had examined its setup or independently verified its results. The authors re-run the three best systems (Talos, Athene, UCLMR) on the FNC-1 data. They show that the official score rewards the dominant and easily recognized unrelated class, so they propose macro-averaged F1 (F1m) instead, which changes the ranking of the top three. A feature ablation informs a new feature-rich stacked LSTM (stackLSTM). It matches the leading systems overall but is better at the rare disagree class. To test generalization, they convert the ARC corpus of user posts into the FNC format and run in-domain and cross-domain experiments. The analysis shows that current models depend mostly on lexical overlap and fail on difficult cases, which calls for methods with deeper semantic understanding. The authors publish their data and code.

Key points

  • The FNC-1 dataset (derived from the Emergent project) contains 75,385 headline–document pairs from 300 topics, split 200/100 by topic for training and testing. Labels are very skewed: UNR 72.8%, DSC 17.7%, AGR 7.4%, DSG 2.0%.
  • On the official FNC metric, TalosComb scores .820, Athene .820 and UCLMR .817. Under F1m the order becomes Athene (.604), UCLMR (.583), TalosComb (.582). The estimated human upper bound is F1m = .754.
  • The proposed stackLSTM reaches the highest F1m (.609) and a DSG F1 of .180, compared with only .035 for TalosComb.
  • The adapted ARC corpus has a Fleiss' κ of .614 and a MACE-based human upper bound of F1m = .773. TalosComb leads in-domain on ARC (F1m .573), while stackLSTM performs best when trained on FNC and tested on ARC (F1m .401).
  • Code and data are available at github.com/UKPLab/coling2018_fake-news-challenge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

65 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1806.05180

@inproceedings{hanselowski2018retrospective,
  title = {{A Retrospective Analysis of the Fake News Challenge Stance-Detection Task}},
  author = {Hanselowski, Andreas and PVS, Avinesh and Schiller, Benjamin and Caspelherr, Felix and Chaudhuri, Debanjan and Meyer, Christian M. and Gurevych, Iryna},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  eprint = {1806.05180},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.05180}
}

References

49 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4):555–596. find
  2. Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). Austin, TX, USA, pages 876–885. find our card
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2001. Latent dirichlet allocation. In Advances in Neural Information Processing Systems 14 (NIPS). Vancouver, BC, Canada, pages 601–608. find our card
  4. Peter Bourgonje, Julian Moreno Schneider, and Georg Rehm. 2017. From Clickbait to Fake News Detection: An Approach based on Detecting the Stance of Headlines to Articles. In Proceedings of the EMNLP 2017 Workshop ‘Natural Language Processing meets Journalism’. Copenhagen, Denmark, pages 84–89. find
  5. Sanjiv R. Das and Mike Y. Chen. 2007. Yahoo! for Amazon: Sentiment Extraction from Small Talk on the Web. Management Science 53(9):1375–1388. find
  6. Richard Davis and Chris Proctor. 2017. Fake News, Real Consequences: Recruiting Neural Networks for the Fight Against Fake News. Online: http://web.stanford.edu/class/cs224n/reports/ 2761239.pdf. Accessed: 2018-03-16. ↗
  7. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American society for information science 41(6):391. find
  8. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval). Vancouver, BC, C… find
  9. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI). Melbourne, Australia, pages 3988–3994. find
  10. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT). San Diego, CA, USA, pages 1163–1168. find
  11. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5):378. find
  12. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  13. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, and Felix Caspelherr. 2017. Description of the system developed by team Athene in the FNC-1, 2017. Online: https://github.com/ hanselowski/athene_system/blob/master/system_description_athene.pdf. Accessed: 2018-03-13. ↗
  14. Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing (IJCNLP). Nagoya, Japan, pages 1348–1356. find
  15. Michiel Hermans and Benjamin Schrauwen. 2013. Training and analysing deep recurrent neural networks. In Advances in neural information processing systems 26 (NIPS). Stateline, NV, USA, pages 190–198. find
  16. Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. In Proceedings of the ICWSM 2017 Workshop on News and Public Opinion. Montréal, QC, Canada, pages 759–766. find
  17. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT). Atlanta, GA, USA, pages 1120–113… find our card
  18. Anderson Jonathan. 1983. Lix and Rix: Variations on a Little-known Readability Index. Journal of Reading 26(6):490–496. find
  19. J. Peter Kincaid, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Research Branch Report 8-75, Naval Technical Training Command, Millingt… find
  20. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research 50:723–762. find
  21. Chih-Jen Lin. 2007. Projected gradient methods for nonnegative matrix factorization. Neural Computation 19(10):2756–2779. find
  22. Michal Lukasik, P.K. Srijith, Duy Vu, Kalina Bontcheva, Arkaitz Zubiaga, and Trevor Cohn. 2016. Hawkes processes for continuous time sequence classification: an application to rumour stance classification in twitter. In Proceedings of 54th Annual Meeting of the Association for Computational Linguist… find
  23. Coleman Mari and Liau Ta Lin. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology 60(2):283. find
  24. G. Harry Mc Laughlin. 1969. SMOG grading—a new readability formula. Journal of reading 12(8):639– 646. find
  25. Rachel McAlpine. 1997. Global English for global business. Longman. find
  26. Philip M. McCarthy. 2005. An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Dissertation Abstracts International 66:12. find
  27. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval). San Diego, CA, USA, pages 31–41. find
  28. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. In Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval). Atlanta, GA, USA, pages 321–327. find
  29. Saif M. Mohammad and Peter D. Turney. 2010. Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon. In Proceedings of the NAACL/HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Los Angeles, CA, USA, pages 26–34. find
  30. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon 29(3):436–465. find
  31. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar, pages 1532–1543. find
  32. Dean Pomerleau and Delip Rao. 2017. The Fake News Challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. http://www.fakenewschallenge.org/. Accessed: 2017-10-20. ↗
  33. Benjamin Riedel, Isabelle Augenstein, Georgios P Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. arXiv preprint arXiv:1707.03264 . find
  34. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment Analysis in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval). Denver, CO, USA, pages 451–463. find
  35. Baird Sean, Sibley Doug, and Pan Yuxi. 2017. Talos Targets Disinformation with Fake News Challenge Victory. http://blog.talosintelligence.com/2017/06/talos-fake-news-challenge. html. Accessed: 2017-12-02. ↗
  36. R.J. Senter and Edgar A. Smith. 1967. Automated readability index. Technical Report AMRL-TR-66-220, University of Cincinnati. find
  37. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter 19(1):22–36. find
  38. Craig Silverman. 2017. Emergent: A real-time rumor tracker. Online: http://www.emergent.info/. Accessed: 2017-12-13. ↗
  39. N. Watson Solomon. 2006. Strain Index: A New Readability Formula. Master thesis, Madurai Kamaraj University. find
  40. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL/HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Los Angeles, CA, USA, pages 116–124. find
  41. Dhanya Sridhar, James R. Foulds, Bert Huang, Lise Getoor, and Marilyn A. Walker. 2015. Joint models of disagreement and stance in online debate. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P… find
  42. Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics 43(3):619–659. find our card
  43. Sanja Štajner, Richard Evans, Constantin Orăsan, and Ruslan Mitkov. 2012. What can readability measures really tell us about text complexity. In Proceedings of the LREC 2012 Workshop on Natural Language Processing for Improving Textual Accessibility (NLP4ITA). Istanbul, Turkey, pages 14–21. find
  44. Mariona Taulé, Maria Antònia Martí, Francisco M. Rangel Pardo, Paolo Rosso, Cristina Bosco, and Viviana Patti. 2017. Overview of the Task on Stance and Gender Detection in Tweets on Catalan Independence. In Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian L… find
  45. James Thorne, Mingjie Chen, Giorgos Myrianthous, Jiashu Pu, Xiaoxuan Wang, and Andreas Vlachos. 2017. Fake news stance detection using stacked ensemble of classifiers. In Proceedings of the EMNLP 2017 Workshop ‘Natural Language Processing meets Journalism’. Copenhagen, Denmark, pages 80– 83. find
  46. Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT). Montréa… find
  47. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP). Vancouver, BC, Canada, pages 347–354. find
  48. Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). San Diego, CA, USA, pages 458–463. find our card
  49. Xiaodan Zhu, Svetlana Kiritchenko, and Saif M. Mohammad. 2014. NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval). Dublin, Ireland, pages 443–447. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/hanselowski-2018-retrospective-analysis-fake-news-challenge/