Elamur Research⌕
☰
home / papers / 2020 / STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval

STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval

Summary

Stance detection (SD) research has mostly focused on short user-generated posts. News articles have received much less attention because few datasets exist, and news SD and social-media SD have developed separately. The authors address this with STANDER, an English news corpus for both SD and fine-grained evidence retrieval (ER). Articles were collected from Factiva and cover four US healthcare mergers and acquisitions. Economics experts assigned each article a stance toward the claim that a given merger will happen. They also marked the text spans that justified their decision, recorded with exact character positions.

An analysis of annotator disagreement over time suggests that ambiguous labels cluster in periods when the deal's outcome was genuinely uncertain. In experiments, current strong models fall well short of human performance, so the corpus works as a hard benchmark. STANDER also targets the same mergers as the WT–WT Twitter SD dataset. Together, the two resources allow the first aligned multi-genre (news plus tweets) study of stance, which is relevant to rumour verification.

Key points

  • The corpus has 3,291 expert-annotated articles on 4 targets (AET_HUM, ANTM_CI, CI_ESRX, CVS_AET), averaging over 800 articles per target. By comparison, the Snopes-based corpus has about 2.22 articles per target.
  • There are four stance labels: support, refute, comment and unrelated. Evidence snippets come with start and end indices, which the authors describe as a first for news SD.
  • Annotation was done by eight Cambridge economists (six PhD students and two lecturers) over 4 months. Each article received 2–4 labels, which were aggregated by majority vote. The most frequent disagreement was between support and comment.
  • State-of-the-art models trail the human upper bound by up to roughly 30%.
  • Daily article volume and WT–WT tweet volume show moderate Spearman correlation: about 0.48–0.63 across all stances, rising to about 0.54–0.65 when only related tweets are counted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.365

@inproceedings{conforti2020stander,
  title = {{STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval}},
  author = {Conforti, Costanza and Berndt, Jakob and Pilehvar, Mohammad Taher and Giannitsarou, Chryssi and Toxvaerd, Flavio and Collier, Nigel},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.365},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.365}
}

References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rakesh Agrawal, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu. 2003. Mining newsgroups using networks arising from social behavior. In Proceedings of the Twelfth International World Wide Web Conference, WWW 2003, Budapest, Hungary, May 20-24, 2003, pages 529–535. ACM. find
  2. Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, Anna Kolliakou, Rob Procter, and Maria Liakata. 2017. Stance classification in out-of-domain rumours: A case study around mental health disorders. In Social Informatics - 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Pro… find
  3. Areej Alhothali and Jesse Hoey. 2015. Good news or bad news: Using affect control theory to analyze readers’ reaction towards news articles. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1548–1… find
  4. Ramy Baly, Mitra Mohtarami, James R. Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages… find
  6. Robert F Bruner and Joseph R Perella. 2004. Applied mergers and acquisitions, volume 173. John Wiley & Sons. find
  7. Lily Canter. 2015. Personalised tweeting: The emerging practices of journalists on twitter. Digital Journalism, 3(6):888–907. find
  8. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175. find our card
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  10. Costanza Conforti, Jakob Berndt, M. Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won’t-they: A very large dataset for stance detection on twitter. In Proceedings of the 2020 Annual Conference of the Association for Computational Linguistics. Association f… find our card
  11. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouve… find
  12. Mark Dredze, Prabhanjan Kambadur, Gary Kazantsev, Gideon Mann, and Miles Osborne. 2016. How twitter is changing the nature of financial news discovery. In Proceedings of the Second International Workshop on Data Science for Macro-Modeling, DSMM@SIGMOD 2016, San Francisco, CA, USA, June 26 - July 1,… find
  13. Marcus Errico, J April, A Asch, L Khalfani, M Smith, and X Ybarra. 1997. The evolution of the summary news lead. Media History Monographs, 1(1). find
  14. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163–1168, San Diego, California. Association for… find
  15. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  16. Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. find
  17. Rumoureval 2019: Determining rumour veracity and support for rumours. CoRR, abs/1809.06683. — [arXiv:1809.06683](https://arxiv.org/abs/1809.06683) find
  18. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018a. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Conference on Computational Linguistics, pages… find our card
  19. Andreas Hanselowski, Avinesh P. V. S., Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018b. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Conference on Computational Linguistics,… find our card
  20. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 493–503, Hong Kong, China. Association for… find
  21. Tomáš Hercig, Peter Krejzl, Barbora Hourová, Josef Steinberger, and Ladislav Lenc. 2017. Detecting stance in czech news commentaries. In Proceedings of the 17th ITAT: Slovenskoceskỳ NLP workshop (SloNLP 2017), volume 1885, pages 176–180. find
  22. Diana Inkpen, Xiaodan Zhu, and Parinaz Sobhani. 2017. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 551–557. Assoc… find our card
  23. Rajiv Johal. 2009. Factiva: Gateway to business information. Journal of Business & Finance Librarianship, 15(1):60–64. find
  24. Michiel Johnson. 2019. Sourcing Twitter: a multi-methodological study on the role of Twitter in economic journalism. Ph.D. thesis, University of Antwerp. find
  25. Michiel Johnson, Steve Paulussen, and Peter Van Aelst. 2018. Much ado about nothing? the low importance of twitter as a sourcing tool for economic journalists. Digital Journalism, 6(7):869–888. find
  26. Manfred Klenner, Don Tuggener, and Simon Clematide. 2017. Stance detection in facebook posts of a german right-wing party. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, LSDSem@EACL 2017, Valencia, Spain, April 3, 2017, pages 31–40. Associa… find
  27. Patty Kostkova, Vino Mano, Heidi J. Larson, and William S. Schulz. 2017. Who is spreading rumours about vaccines? influential user impact modelling in social networks. In Proceedings of the 2017 International Conference on Digital Health, DH ’17, page 48–52, New York, NY, USA. Association for Comput… find
  28. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1). find
  29. Anders Edelbo Lillie and Emil Refsgaard Middelboe. 2019. Fake news detection using stance classification: A survey. CoRR, abs/1907.00181. — [arXiv:1907.00181](https://arxiv.org/abs/1907.00181) find
  30. Sung Hoon Lim and Conrad S. Tucker. 2019. Mining twitter data for causal links between tweets and real-world outcomes. Expert Syst. Appl. X, 3:100007. find
  31. Edward Loper and Steven Bird. 2002. Nltk: the natural language toolkit. arXiv preprint cs/0205028. find
  32. Wes McKinney. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference, pages 56 – 61. find
  33. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. 2016. A dataset for detecting stance in tweets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016. European Language Resources… find
  34. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Trans. Internet Techn., 17(3):26:1–26:23. find
  35. Claudia Orellana-Rodriguez and Mark T. Keane. 2018. Attention to news and its dissemination on twitter: A survey. Computer Science Review, 29:74 – 94. find
  36. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  37. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  38. Jenifer Piesse, Cheng-Few Lee, Lin Lin, and Hsien-Chang Kuo. 2013. Merger and acquisition: Definitions, motives, and market responses. Encyclopedia of Finance, pages 411–420. find
  39. Dean Pomerleau and Delip Rao. 2017. Fake news challenge. find
  40. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, April 3-7, 2017, pages… find
  41. Horst Pöttker. 2003. News and its communicative quality: The inverted pyramid—when and why did it appear? Journalism Studies, 4(4):501–511. find
  42. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 33… find our card
  43. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  44. Benjamin Riedel, Isabelle Augenstein, Georgios P Sp-ithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. arXiv preprint arXiv:1707.03264. find
  45. Md Main Uddin Rony, Mohammad Yousuf, and Naeemul Hassan. 2018. A large-scale study of social media sources in news articles. arXiv preprint arXiv:1810.13078. — [arXiv:1810.13078](https://arxiv.org/abs/1810.13078) find
  46. Christopher Scanlan. 2000. Reporting and writing: Basics for the 21st century. Oxford University Press. find
  47. Maria Skeppstedt, Andreas Kerren, and Manfred Stede. 2017. Automatic detection of stance towards vaccination in online discussion forums. In Proceedings of the International Workshop on Digital Disease Detection using Social Media, DDDSM@IJCNLP 2017, Taipei, Taiwan, November 27, 2017, pages 1– 8. find
  48. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective lstms for target-dependent sentiment classification. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages 3298– 3307. A… find
  49. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 3346–3359. Association for Computatio… find
  50. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 find our card
  51. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 809–819. Association for Computational Linguistics. find
  52. Jannis Vamvas and Rico Sennrich. 2020. X-stance: A multilingual multi-target dataset for stance detection. CoRR, abs/2003.08385. — [arXiv:2003.08385](https://arxiv.org/abs/2003.08385) find
  53. Sarah Van Leuven and Annelore Deprez. 2017. ‘to follow or not to follow?’: How belgian health journalists use twitter to monitor potential sources. Journal of Applied Journalism & Media Studies, 6(3):545– 566. find
  54. Cohn, and Isabelle Augenstein. 2018b. Discourse-aware rumour stance classification in social media using sequential classifiers. Inf. Process. Manage., 54(2):273–290. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/conforti-2020-stander-expert-annotated-dataset-news/