Elamur Research⌕
☰
home / papers / 2020 / Stance Detection Benchmark: How Robust Is Your Stance Detection?

Stance Detection Benchmark: How Robust Is Your Stance Detection?

Paper · 2020 · Benjamin Schiller, Johannes Daxenberger and Iryna Gurevych

Summary

Stance Detection (StD) means deciding whether a piece of text, such as a tweet, post, claim or news article, supports or opposes a given topic or claim. It serves as a foundation for tasks like spotting fake news, verifying claims and retrieving arguments. Humans agree on stance far more often than models predict it correctly. StD datasets also differ widely in domain, input format, label set and size. The authors therefore build a benchmark of ten StD datasets drawn from five domains. They train them jointly in a multi-dataset learning (MDL) setup and add transfer learning (TL) from related tasks, adapting the MT-DNN framework. Both techniques raise accuracy on average, and the combined model sets new state-of-the-art results on five datasets. However, simple adversarial attacks (spelling errors, negation, paraphrasing) hurt the MDL model more than single-dataset (SDL) BERT models. Low-resource experiments suggest the cause is overfitting to dataset biases when training data is abundant. The work argues that multi-task NLP needs de-biasing methods and releases the full benchmark for reuse.

Key points

  • The benchmark covers ten corpora, including arc, argmin, fnc1, iac1, ibmcs, perspectrum, scd, semeval2016t6, semeval2019t7 and snopes. Their training sizes range from 2,394 to 75,385 samples.
  • Average gains are about +4 pp from MDL and +3.4 pp from TL. The gap to human agreement remains large, for example FNC-1 at 61.1% versus 75.4% inter-annotator agreement.
  • Robustness is measured with Resilience_rel, a modified version of the Resilience score of Thorne et al. (2019) that also accounts for clean test-set performance. With full training data, the overall score is 92.9% for MT-DNN MDL versus 96.7% for BERT SDL.
  • Shrinking the training data to 10% narrows this robustness gap from 3.8 pp to 1.5 pp, mainly on the negation and paraphrase attacks. At the same time, MDL's test-score advantage over SDL grows from .0514 to .0953.
  • Data splits, models and code are available at github.com/UKPLab/mdl-stance-robustness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany114 works · 36,326 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Benjamin Schiller, Johannes Daxenberger and Iryna Gurevych. 2020. Stance Detection Benchmark: How Robust Is Your Stance Detection?. arXiv:2001.01565.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.01565 · arXiv 2001.01565

@misc{schiller2020stance,
  title = {{Stance Detection Benchmark: How Robust Is Your Stance Detection?}},
  author = {Benjamin Schiller, Johannes Daxenberger and Iryna Gurevych},
  year = {2020},
  doi = {10.48550/arxiv.2001.01565},
  eprint = {2001.01565},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.01565}
}

References

49 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din-uzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In EACL’17, pages 251–261.
  2. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173. find
  3. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle:discovering diverse perspectives about claims. In NAACL’19, pages 542–557.
  4. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In EMNLP-IJCNLP’19, pages 4067–4080. find
  5. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. arXiv preprint arXiv:1704.05972.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  7. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In NAACL’16, pages 1163–1168.
  8. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In EMNLP-IJCNLP’19, pages 1161–1166. find
  9. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In ACL’18 (Volume 2: Short Papers), pages 650–655.
  10. Genevieve Gorrell, Ahmet Aker, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2019. Semeval-2019 task 7: Rumoureval, determining rumour veracity and support for rumours. In SemEval-2019, pages 845–854.
  11. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In NAACL’18, pages 1930– 1940.
  12. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In COLING’18, pages 1859–1874.
  13. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In CoNLL’19, pages 493–503.
  14. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In IJCNLP’13, pages 1348– 1356.
  15. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In DeepLo’19, pages 132–142.
  16. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. arXiv preprint arXiv:1704.07431.
  17. Yan Jiang. 2019. Using machine learning for stance detection. Master’s thesis, The University of Texas at Austin, 1. find
  18. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Institute for Simulation and Training, University of Central Florida. find
  19. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In ACL’17, pages 67–72.
  20. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In ACL’19, pages 4487–4496.
  21. Taylor Mahler, Willy Cheung, Micha Elsner, David King, Marie-Catherine de Marneffe, Cory Shain, Symon Stevens-Guille, and Michael White. 2017. Breaking NLP: Using Morphosyntax, Semantics, Pragmatics and World Knowledge to Fool Sentiment Analysis Systems. EMNLP’17, pages 33–39.
  22. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In EACL’17, pages 881–893.
  23. Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural nli models to integrate logical background knowledge. In CoNLL’18, pages 65–74. find
  24. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In SemEval-2016, pages 31–41.
  25. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In COLING’18, pages 2340–2353. find
  26. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In ACL’19, pages 4658–4664. find
  27. Dean Pomerleau and Delip Rao. 2017. The Fake News Challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. http://www.fakenewschallenge.org/. [Online; accessed 06-January-2020]. ↗
  28. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In WWW’17, pages 1003–1012. find
  29. Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2019. STANCY: Stance classification based on consistency cues. In EMNLP-IJCNLP’19, pages 6412–6417. find
  30. Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019. Sampling bias in deep active classification: An empirical study. In EMNLP-IJCNLP’19, pages 4056–4066. find
  31. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. Combating adversarial misspellings with robust word recognition. In ACL’19, pages 5582– 5591.
  32. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
  33. Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian optimization. In EMNLP’17, pages 372–382.
  34. Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych. 2018. Multi-task learning for argumentation mining in low-resource settings. In NAACL’18, pages 35–41. find
  35. Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In EACL’17, pages 551–557.
  36. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In NAACL-HLT’10, pages 116–124.
  37. Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018. Cross-topic argument mining from heterogeneous sources. In EMNLP’18, pages 3664–3674.
  38. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In EMNLP’06, pages 327–335.
  39. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In EMNLP-IJCNLP’19, pages 2944–2953. find
  40. Marilyn A Walker, Pranav Anand, Rob Abbott, Jean E Fox Tree, Craig Martell, and Joseph King. 2012a. That is your evidence?: Classifying stance in online political debate. Decision Support Systems, 53(4):719–729. find
  41. Marilyn A Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012b. A corpus for research on deliberation and debate. In LREC’12, pages 812–817.
  42. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP’18 Workshop BlackboxNLP, pages 353– 355.
  43. Derek Wang, Chaoran Li, Sheng Wen, Yang Xiang, Wanlei Zhou, and Surya Nepal. 2018b. Defensive collaborative multi-task training-defending against adversarial attack towards deep neural networks. arXiv preprint arXiv:1803.05123. — [arXiv:1803.05123](https://arxiv.org/abs/1803.05123) find
  44. Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. In NAACL’18, pages 575–581. find
  45. Penghui Wei, Wenji Mao, and Daniel Zeng. 2018. A target-guided neural memory model for stance detection in twitter. In IJCNN’18, pages 1–8. find
  46. Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. Journal of Big data, 3(1):9. find
  47. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  48. Ruoyao Yang, Wanying Xie, Chunhua Liu, and Dong Yu. 2019. BLCU NLP at SemEval-2019 task 7: An inference chain-based GPT model for rumour evaluation. In SemEval-2019, pages 1090–1096. find
  49. Qiang Zhang, Emine Yilmaz, and Shangsong Liang. 2018. Ranking-based method for news stance detection. In WWW’18, pages 41–42. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schiller-2020-stance-detection-benchmark-robust-stance/