Elamur Research⌕
☰
home / papers / 2021 / Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study

Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study

Summary

Cross-topic stance detection asks a model to label a sentence as pro, con, or non-argumentative toward a debate topic it never saw during training. The authors revisit the BERT-based approach of Reimers et al. (2019) and reproduce its classification experiments on the UKP Sentential Argument Mining Corpus. They train on seven topics and evaluate on the held-out eighth. Following the reproduction framework of Cohen et al. (2018), they check whether numeric values and findings match, repeating fine-tuning across several random seeds. The original numbers are largely recovered.

Breaking results down by topic and by class tells a less uniform story. Topic, class, and their interaction all affect scores, and a simple tf-idf SVM competes with BERT on certain topics. A lexical feature analysis suggests that stance-bearing vocabulary is tightly bound to individual debates.

The study argues that a single score averaged over topics conceals these weaknesses. Future cross-topic models should therefore account for topic-specific wording and the socio-cultural background of debates.

Key points

  • Data: the UKP dataset (Stab et al., 2018) contains 25,492 sentences from 400 web texts covering 8 topics:
    • school uniforms, the minimum wage, legalizing marijuana, nuclear energy, the death penalty, cloning, gun control, and abortion
    • labels are pro, con, and no argument
    • the IBM Debater dataset was excluded because it lacks pro/con labels.
  • Reproduction criterion: a value counts as reproduced if the original result falls within two standard deviations of the mean over 10 random seeds. The argument-clustering part of the original paper was not replicated.
  • Per-topic results: cross-topic BERT scores below its average on the pro and con classes for some topics, notably abortion and gun control. On gun control it shows error patterns similar to earlier LSTM models.
  • SVM feature analysis: a tf-idf SVM reaches F1 = .573 trained on all topics and .595 on abortion alone. Their most informative features barely overlap, which points to topic-specific stance vocabulary (e.g., "babies", "birth", "conception" for abortion).
  • Recommendation: evaluate cross-topic models per topic, or by the worst-performing topic, rather than only by the average. Topic diversity beyond U.S.-centric debates is also needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.argmining-1.5 · arXiv 2110.07693

@inproceedings{reuver2021is,
  title = {{Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study}},
  author = {Reuver, Myrthe and Verberne, Suzan and Morante, Roser and Fokkens, Antske},
  booktitle = {Proceedings of the 8th Workshop on Argument Mining},
  year = {2021},
  doi = {10.18653/v1/2021.argmining-1.5},
  eprint = {2110.07693},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.argmining-1.5}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeer ALDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597. find
  2. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163–1168. find
  3. Emily Allaway, Malavika Srikanth, and Kathleen McKeown. 2021. Adversarial learning for zero-shot stance detection on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4756–4767, Onlin… find our card
  4. Antske Fokkens, Marieke Van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from reproduction problems: What replication failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1691–… find
  5. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 381–393, Online.… find
  6. Andreas Hanselowski, PVS Avinesh, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1… find our card
  7. Emily Bender. 2019. The \#benderrule: On naming the languages we study and why it matters. The Gradient. find
  8. Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021. A survey on stance detection for mis-and disinformation identification. arXiv preprint arXiv:2103.00242. find our card
  9. Mihaela Bošnjak and Mladen Karan. 2019. Data set for stance and sentiment analysis from user comments on croatian news. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 50–55. find
  10. Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. Covidlies: Detecting covid-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. find
  11. K Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany J Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurélie Névéol, Cyril Grouin, and Lawrence E Hunter. 2018. Three dimensions of reproducibility in natural language processing. In LREC... International Conference on Language Resources & Ev… find
  12. Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2010. Vocabulary choice as an indicator of perspective. In Proceedings of the ACL 2010 conference short papers, pages 253–257. find
  13. Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020. Unsupervised user stance detection on twitter. In Proceedings of the Interna-Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1). find
  14. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Ling… find
  15. Conference on Natural Language Processing of the AFNLP, pages 226–234, Suntec, Singapore. Association for Computational Linguistics. find
  16. Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018. Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3664–3674, Brussels, Belgium. Association for Computationa… find
  17. Stefan Ollinger, Lorik Dumani, Premtim Sahitaj, Ralph Bergmann, and Ralf Schenkel. 2020. Same Side Stance Classification Task: Facilitating Argument Stance Classification by Fine-tuning a BERT Model. arXiv:2004.11163 [cs]. ArXiv: 2004.11163. — [arXiv:2004.11163](https://arxiv.org/abs/2004.11163) find
  18. Benno Stein, Yamen Ajjour, Roxanne El Baff, Khalid Al-Khatib, Philipp Cimiano, AG Semantic Computing, and Henning Wachsmuth. 2021. Same side stance classification. Preprint. find
  19. Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and Clustering of Arguments with Contextualized Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567– 578, Fl… find
  20. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP ’06, page 327–335, USA. Association for Computational Linguis… find
  21. Myrthe Reuver, Antske Fokkens, and Suzan Verberne. 2021. No NLP task should be an island: Multi-disciplinarity for diversity in news recommender systems. In Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, pages 45–55, Online. Association for Computat… find
  22. Terne Sasha Thorn Jakobsen, Maria Barrett, and Anders Søgaard. 2021. Spurious correlations in cross-topic argument mining. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 263–277, Online. Association for Computational Linguistics. find
  23. Benjamin Riedel, Isabelle Augenstein, Georgios P Sp-ithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. arXiv preprint arXiv:1707.03264. find
  24. Penghui Wei and Wenji Mao. 2019. Modeling transferable topics for cross-target stance detection. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1173–1176. find
  25. Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2021. Stance Detection Benchmark: How Robust is Your Stance Detection? KI - Künstliche Intelligenz. find
  26. Lixing Zhu, Yulan He, and Deyu Zhou. 2019. Hierarchical viewpoint discovery from tweets using bayesian modelling. Expert Systems with Applications, 116:430–438. find
  27. David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Paper… find
  28. Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computationa… find
  29. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint find

↑ back to top