Elamur Research
☰
home / papers / 2020 / Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations

Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.717.pdf

Summary

Stance detection decides whether a text is for, against or neutral toward a topic. Most existing systems need labelled examples for each topic, or a person has to explain how a new topic relates to the training topics. Allaway and McKeown instead study zero-shot and few-shot settings, where models are tested on many topics they saw rarely or never during training. To support this, they build VAST (VAried Stance Topics) from New York Times "Room for Debate" comments in the ARC corpus. Topics were first extracted automatically, then fixed or expanded by Amazon Mechanical Turk workers, who also labelled stance. Extra neutral examples were made by pairing comments with unrelated topics. The resulting topics cover politics, education and public health, and the same topic often appears under several wordings. The authors also propose TGA Net, which clusters contextual embeddings to get generalized topic representations. These capture how topics relate to each other without any human input. Compared with a BERT-joint baseline, TGA Net scores clearly better on pro labels and about the same on the others. It also handles sarcasm and quotes better and depends less on misleading sentiment words.

Key points

  • VAST uses the topic-phrase definition of stance (pro/con/neutral). It has three kinds of labels: Heur (heuristically extracted topics), Corr (topics corrected by annotators) and List (topics listed by annotators).
  • The automatic topic extraction step produced 304 unique topics from 3,365 comments. Crowdsourcing yielded only 350 neutral labels, so the authors generated more neutral examples by pairing comments with unrelated topics.
  • Annotators agreed moderately: Krippendorff α was 0.427 and raw agreement was 75%. MACE was used to filter out low-quality workers.
  • Stance and sentiment often diverge in the test set. 80% of pro examples contain mostly positive words, but only 41% of con examples contain mostly negative ones. When sentiment words were swapped for WordNet synonyms of opposite polarity, TGA Net's scores shifted less than BERT-joint's.
  • The dataset and models are public at github.com/emilyallaway/zero-shot-stance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

104 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.717 · arXiv 2010.03640

@inproceedings{allaway2020zero,
  title = {{Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations}},
  author = {Allaway, Emily and McKeown, Kathleen},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.717},
  eprint = {2010.03640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.717}
}

References

39 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn A. Walker. 2016. Internet argument corpus 2.0: An sql schema for dialogic social media and the corpora to go with it. In LREC. find
  2. Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In EMNLP. find our card
  3. Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din-uzzo, Amrita Saha, and N. Slonim. 2017. Stance classification of context-dependent claims. In EACL. find
  4. Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2010. Vocabulary choice as an indicator of perspective. In Proceedings of the ACL 2010 Conference Short Papers, pages 253–257, Uppsala, Sweden. Association for Computational Linguistics. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805. find
  6. Marcelo Dias and Karin Becker. 2016. Inf-ufrgs-opinion-mining at semeval-2016 task 6: Automatic generation of a training corpus for unsupervised identification of stance in tweets. In SemEval@NAACL-HLT. find
  7. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In EMNLP/IJCNLP. find
  8. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention. In IJCAI. find
  9. Adam Faulkner. 2014. Automated classification of stance in student essays: An approach using stance target information and the wikipedia link-based measure. In FLAIRS Conference. find
  10. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In HLT-NAACL. find
  11. Swapna Gottipati, Minghui Qiu, Yanchuan Sim, Jing Jiang, and Noah A. Smith. 2013. Learning topics and positions from debatepedia. In EMNLP. find
  12. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In NAACL-HLT. find
  13. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In IJCNLP. find
  14. Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In EMNLP. find
  15. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with mace. In HLT-NAACL. find our card
  16. Diederik P. Kingma and Jimmy Ba. 2015. A method for stochastic optimization. abs/1412.6980. find
  17. Adam: CoRR, Peter Krejzl, Barbora Hourová, and Josef Steinberger. 2017. Stance detection in online discussions. ArXiv, abs/1701.00504. — [arXiv:1701.00504](https://arxiv.org/abs/1701.00504) find
  18. Dilek Küçük. 2017. Stance detection in turkish tweets. ArXiv, abs/1706.06894. — [arXiv:1706.06894](https://arxiv.org/abs/1706.06894) find
  19. Dilek Küçük and F. Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53:1 – 37. find
  20. Chang Li, Aldo Porco, and Dan Goldwasser. 2018. Structured representation learning for online debate stance prediction. In COLING. find
  21. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander G. Hauptmann. 2006. Which side are you on? identifying perspectives at the document and sentence levels. In CoNLL. find
  22. Nikita Lozhnikov, Leon Derczynski, and Manuel Maz-zara. 2018. Stance prediction for russian: Data and analysis. ArXiv, abs/1809.01574. — [arXiv:1809.01574](https://arxiv.org/abs/1809.01574) find
  23. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In SemEval@NAACL-HLT. find
  24. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Trans. Internet Techn., 17:26:1–26:23. find
  25. Akiko Murakami and Raymond H. Putra. 2010. Support or oppose? classifying positions in online debates from reply activities and opinion expressions. In COLING. find
  26. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  27. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In HLT-NAACL 2010. find
  28. Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, and Marilyn Walker. 2015. Joint models of disagreement and stance in online debate. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Process… find
  29. Mariona Taulé, Maria Antònia Martı́, Francisco M. Rangel Pardo, Paolo Rosso, Cristina Bosco, and Viviana Patti. 2017. Overview of the task on stance and gender detection in tweets on catalan independence. In IberEval@SEPLN. find
  30. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In NAACL-HLT. find our card
  31. Adam Tsakalidis, Nikolaos Aletras, Alexandra I. Cristea, and Maria Liakata. 2018. Nowcasting the stance of social media users in a sudden vote: The case of the greek referendum. Proceedings of the 27th ACM International Conference on Information and Knowledge Management. find
  32. Jannis Vamvas and Rico Sennrich. 2020. X -stance: A multilingual multi-target dataset for stance detection. ArXiv, abs/2003.08385. — [arXiv:2003.08385](https://arxiv.org/abs/2003.08385) find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Thirty-First Annual Conference on Neural Information Systems. find
  34. Marilyn A. Walker, Jean E. Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In LREC. find
  35. Jr. Joe H. Ward. 1963. Hierarchical grouping to optimize an objective function. find
  36. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at semeval-2016 task 6 : A specific convolutional neural network system for effective stance detection. In SemEval@NAACL-HLT. find
  37. Theresa Wilson, Janyce Wiebe, and Claire Cardie. 2017. Mpqa opinion corpus. find
  38. Chang Xu, Cécile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In ACL. find
  39. Guido Zarrella and Amy Marsh. 2016. Mitre at semeval-2016 task 6: Transfer learning for stance detection. In SemEval@NAACL-HLT. find