Elamur Research⌕
☰
home / papers / 2017 / A Dataset for Multi-Target Stance Detection

A Dataset for Multi-Target Stance Detection

Summary

Most stance classifiers judge an author's position toward each target separately. However, attitudes toward related targets often depend on each other, such as rival candidates in an election. Sobhani, Inkpen and Zhu address this gap with the Multi-Target Stance Dataset. It contains 4,455 tweets about the 2016 US presidential race, each labeled for stance toward two candidates at once. Annotations were crowdsourced on CrowdFlower with gold-question quality control, and items with weak agreement were filtered out. The authors also release a large pool of unlabeled election tweets. As a modeling approach, they frame the task as sequence-to-sequence learning. A bidirectional GRU encoder reads the tweet, and an attention-based decoder outputs one stance label per target, with each label conditioned on the ones before it. This attention-based Seq2Seq model beats several alternatives on macro-averaged F-score: independent SVMs, window-based SVMs, cascading SVMs and a single 9-class SVM. The differences are statistically significant. The work offers the first reasonably sized benchmark for joint stance modeling and evidence that modeling dependencies between targets helps.

Key points

  • The dataset covers three target pairs: Clinton–Sanders (1,366 tweets), Clinton–Trump (1,722) and Cruz–Trump (1,317). It is split by timestamp into 70/10/20 portions, giving 3,119 train, 446 dev and 890 test tweets.
  • Every tweet received at least eight annotations. Tweets with under 50% agreement on either target were dropped, and the inter-annotator agreement is 79.74%.
  • About 20% of tweets show the same stance toward both candidates, about 50% show opposing stances, and in about 17% neither stance can be inferred.
  • Macro-F results (average F1 of Favor/Against):
    • Baselines: random 34.26, majority 32.11.
    • SVM variants: independent 51.37, window-based 48.32, cascading 52.05, 9-class 50.63.
    • Seq2Seq: 54.81.
  • Model setup:
    • Encoder and decoder use GRU units with 128 cells, trained with Adadelta.
    • Word embeddings have 100 dimensions and were pretrained with Word2Vec Skip-gram on roughly 11.9M unlabeled tweets.
    • Tokenization relied on the CMU Twitter NLP tool.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

59 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 46 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sobhani2017dataset,
  title = {{A Dataset for Multi-Target Stance Detection}},
  author = {Sobhani, Parinaz and Inkpen, Diana and Zhu, Xiaodan},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.18653/v1/e17-2088},
  url = {https://doi.org/10.18653/v1/e17-2088}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Portland, Oregon, USA, June. Association for Computational Linguistics. find
  2. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 1–9, Portla… find
  3. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. find our card
  4. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348–1356, Nagoya, Japan, October. Asian Federation of Natural Language Proc… find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  6. Tomáš Brychcı́n, Michal Konkol, and Josef Steinberger. 2014. Uwb: Machine learning approach to aspect-based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 817–822, Dublin, Ireland, August. Association for Computational Linguistics… find
  7. Geremy Heitz, Stephen Gould, Ashutosh Saxena, and Daphne Koller. 2009. Cascaded classification models: Combining models for holistic scene understanding. In Advances in Neural Information Processing Systems, pages 641–648. find
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  9. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. find our card
  10. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  11. Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 720–728, Doha, Qatar, October. Association for Computational Linguistics. find
  12. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. End-to-end continuous speech recognition using attention-based recurrent nn: first results. arXiv preprint arXiv:1412.1602. — [arXiv:1412.1602](https://arxiv.org/abs/1412.1602) find
  13. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. find
  14. Lingjia Deng and Janyce Wiebe. 2015a. Joint prediction for entity/event-level sentiment analysis using probabilistic soft logic models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 179–189, Lisbon, Portugal, September. Association for Computational… find
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  16. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204–2212. find
  17. Lingjia Deng and Janyce Wiebe. 2015b. Mpqa 3.0: An entity/event-level sentiment corpus. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1323–1328, Denver, Colorado, May–June. Association for Comp… find
  18. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41, San Diego, California, June. Association for Computationa… find
  19. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Assoc… find
  20. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2016c. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, In Press. find
  21. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164. find our card
  22. Fabian Pedregosa, Gaël Varoquaux, et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  23. Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012a. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 592–596, Montréal… find
  24. Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From argumentation mining to stance classification. In Proceedings of the Workshop on Argumentation Mining, pages 67–77, Denver, Colorado, USA. find
  25. Marilyn Walker, Grace Lin, and Jennifer Sawyer. 2012b. An annotated corpus of film dialogue for learning and characterizing character style. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proce… find
  26. Parinaz Sobhani, Saif M. Mohammad, and Svetlana Kiritchenko. 2016. Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*Sem), Edinburgh, Scotland. find
  27. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  28. Matthew D. Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  29. Xiaodan Zhu, Hongyu Guo, and Parinaz Sobhani. 2015a. Neural networks for integrating compositional and non-compositional sentiment in sentiment composition. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 1–9, Denver, Colorado, June. Association for Comput… find
  30. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 226–234, Suntec, Singapore. find
  31. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015b. Long short-term memory over recursive structures. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 1604– 1612. find
  32. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124, Los Angeles, CA, June. Association for Computational Linguistics. find
  33. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2016. Dag-structured long short-term memory for semantic compositionality. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 917–926, San Diego, Califo… find
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  35. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 327–335, Sydney, Australia, July. Association for Computation… find
  36. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015a. Grammar as a foreign language. In Advances in Neural Information Processing Systems, pages 2773–2781. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sobhani-2017-dataset-multi-target-stance-detection/