Elamur Research⌕
☰
home / papers / 2016 / Stance and Sentiment in Tweets

Stance and Sentiment in Tweets

Paper · 2016 · Saif M. Mohammad, Parinaz Sobhani and Svetlana Kiritchenko

Summary

This paper treats stance detection as a separate problem from sentiment analysis. Stance detection asks whether a tweet's author supports or opposes a given target. That target may not be named in the tweet, and the tweet's opinion may be about something else. The same stance can also be expressed with positive or negative wording, but before this work no corpus labeled both stance and sentiment.

The authors built the first such resource. It contains more than 4,000 tweets paired with five targets familiar to US audiences. The tweets were gathered through query hashtags and labeled by crowdworkers for three things: stance, whether the opinion is aimed at the given target or another entity, and sentiment polarity. The data was used for SemEval-2016 Task 6 and comes with an interactive visualizer.

The authors also propose a simple linear-kernel SVM. It combines word and character n-grams, sentiment-lexicon features and word embeddings learned from unlabeled tweets, and it beats every shared-task submission. Their experiments show that sentiment helps predict stance but cannot determine it on its own. This makes the dataset a useful benchmark for opinion mining beyond simple polarity.

Key points

  • The five targets are Hillary Clinton, the Feminist Movement, Atheism, the Legalization of Abortion, and the view that Climate Change is a Real Concern.
  • SemEval-2016 Task 6 used 2,914 training and 1,249 test instances and drew entries from 19 teams. The winning system (Zarrella and Marsh) used two RNNs, one pre-trained on hashtag prediction, and reached an average F-score of 67.8.
  • The authors' SVM reaches an F-score of 70.3 on the same test set, using distant supervision and embeddings from additional unlabeled data.
  • An oracle that knows the gold sentiment and target-of-opinion labels predicts stance with an F-score of only 59.6. Sentiment features also help sentiment classification much more than stance classification.
  • When a tweet's opinion concerns an entity other than the target, the automatic system performs close to the majority baseline. Humans can still infer stance in these cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Research Council CanadaOttawa · Canada60 works · 21,790 citations🎓University of OttawaOttawa · Canada13 works · 2,514 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Saif M. Mohammad, Parinaz Sobhani and Svetlana Kiritchenko. 2016. Stance and Sentiment in Tweets. arXiv:1605.01655.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1605.01655 · arXiv 1605.01655

@misc{mohammad2016stance,
  title = {{Stance and Sentiment in Tweets}},
  author = {Saif M. Mohammad, Parinaz Sobhani and Svetlana Kiritchenko},
  year = {2016},
  doi = {10.48550/arxiv.1605.01655},
  eprint = {1605.01655},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.01655}
}

References

69 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the Workshop on Computational Approaches to Subjectivity and Sentiment Analysis. 1–9.
  2. Ion Androutsopoulos and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research (2010), 135–187.
  3. Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In Advances in Neural Information Processing Systems.
  4. Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa Dang, and Danilo Giampiccolo. 2011. The seventh PASCAL recognizing textual entailment challenge. In Proceedings of Text Analysis Conference. find
  5. Adam Bermingham and Alan Smeaton. 2011. On Using Twitter to Monitor Political Sentiment and Predict Election Results. In Proceedings of the Workshop on Sentiment Analysis where AI meets Psychology. Chiang Mai, Thailand, 2–10. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the International Conference on Machine Learning. 160–167. find
  7. Michael Conover, Bruno Gonçalves, Jacob Ratkiewicz, Alessandro Flammini, and Filippo Menczer. 2011a. Predicting the Political Alignment of Twitter Users. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust. 192–199. find
  8. Michael Conover, Jacob Ratkiewicz, Matthew R. Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011b. Political Polarization on Twitter. In Proceedings of the International AAAI Conference on Weblogs and Social Media. 89–96. find
  9. Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Proceedings of the Conference on Intelligent Systems for Molecular Biology. 77–86. find
  10. Ido Dagan and Oren Glickman. 2004. Probabilistic textual entailment: Generic applied modeling of language variability. In PASCAL workshop on Text Understanding and Mining. 26–29. find
  11. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Morgan & Claypool Publishers. find
  12. Lingjia Deng, Janyce Wiebe, and Yoonjung Choi. 2014. Joint inference and disambiguation of implicit sentiments via implicature constraints. In Proceedings of the International Conference on Computational Linguistics. 79–88.
  13. Sarah Djemili, Julien Longhi, Claudia Marinica, Dimitris Kotzinos, and Georges-Elia Sarfati. 2014. What does Twitter have to say about ideology?. In Proceedings of the Natural Language Processing for Computer-Mediated Communication/Social Media-Pre-conference workshop at Konvens. find
  14. Myroslava O. Dzikovska, Rodney D. Nielsen, and Claudia Leacock. 2016. The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications. Language Resources and Evaluation 50 (2016), 67–93. Issue 1. find
  15. Adam Faulkner. 2014. Automated Classification of Stance in Student Essays: An Approach Using Stance Target Information and the Wikipedia Link-Based Measure. In Proceedings of the Flairs Conference. find
  16. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of the Annual Meeting of the Associatio…
  17. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification using Distant Supervision. Technical Report. Stanford University. find
  18. Jennifer Golbeck and Derek Hansen. 2011. Computing Political Preference Among Twitter Followers. In Proceedings of the Conference on Human Factors in Computing Systems. New York, NY, 1105–1108. find
  19. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proceedings of the International Joint Conference on Natural Language Processing. 1348–1356.
  20. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 168–177. find
  21. Stephanie D. Husby and Denilson Barbosa. 2012. Topic classification of blog posts using distant supervision. In Proceedings of the Workshop on Semantic Analysis in Social Media. 28–36. find
  22. ACM Transactions on Embedded Computing Systems, Vol. 0, No. 0, Article 0, Publication date: 2016. find
  23. Yoshikiyo Kato, Sadao Kurohashi, Kentaro Inui, Robert Malouf, and Tony Mullen. 2008. Taking sides: User classification for informal online political discourse. Internet Research 18, 2 (2008), 177–190. find
  24. Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. 2014b. NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the International Workshop on Semantic Evaluation. Dublin, Ireland.
  25. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014a. Sentiment Analysis of Short Informal Texts. Journal of Artificial Intelligence Research 50 (2014), 723–762. find
  26. Florian Kunneman, Christine Liebrecht, and Antal van den Bosch. 2014. The (un)predictability of emotional hashtags in Twitter. In Proceedings of the Workshop on Language Analysis for Social Media. 26–34. find
  27. Vasileios Lampos, Daniel Preotiuc-Pietro, and Trevor Cohn. 2013. A user-centric model of voting intention from social media. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 993–1003. find
  28. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proceedings of the Conference on Computational Natural Language Learning. 109–116. find
  29. Bing Liu. 2015. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press. find
  30. Bing Liu and Lei Zhang. 2012. A Survey of Opinion Mining and Sentiment Analysis. In Mining Text Data, Charu C. Aggarwal and ChengXiang Zhai (Eds.). Springer, 415–463. find
  31. Walid Magdy, Hassan Sajjad, Tarek El-Ganainy, and Fabrizio Sebastiani. 2015. Bridging social media via distant supervision. Social Network Analysis and Mining 5, 1 (2015), 1–12. find
  32. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 Task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the International…
  33. Diana Maynard and Adam Funk. 2011. Automatic Detection of Political Opinions in Tweets. In Proceedings of the ESWC Workshop on the Semantic Web. 88–99. find
  34. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111–3119.
  35. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 1003–101…
  36. Andriy Mnih and Geoffrey E. Hinton. 2009. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems. 1081–1088. find
  37. Saif M. Mohammad. 2012. \#Emotional Tweets. In Proceedings of the Joint Conference on Lexical and Computational Semantics. Montréal, Canada, 246–255.
  38. Saif M Mohammad. 2015. Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text. (2015). find
  39. Saif M. Mohammad. 2016. A Practical Guide to Sentiment Annotation: Challenges and Solutions. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. find
  40. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the International Workshop on Semantic Evaluation. San Diego, California.
  41. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the State-ofthe-Art in Sentiment Analysis of Tweets. In Proceedings of the International Workshop on Semantic Evaluation. Atlanta, Georgia, USA. find
  42. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. 26–34.
  43. Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing and Management 51 (2015), 480–499. find
  44. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2, 1–2 (2008), 1–135. find
  45. Ankur P. Parikh, Hoifung Poon, and Kristina Toutanova. 2015. Grounded Semantic Parsing for Complex Knowledge Extraction. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  46. ACM Transactions on Embedded Computing Systems, Vol. 0, No. 0, Article 0, Publication date: 2016. find
  47. Fabian Pedregosa, Gaël Varoquaux, and others. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830. find
  48. Maria Pontiki, Dimitrios Galanis, Haris Papageogiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 Task 12: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation. Denver, Colorado.
  49. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. In Proceedings of the International Workshop on Semantic Evaluation. Dublin, Ireland.
  50. Ashwin Rajadesingan and Huan Liu. 2014. Identifying Users with Opposing Opinions in Twitter Debates. In Proceedings of the Conference on Social Computing, Behavioral-Cultural Modeling and Prediction. Washington, DC, USA, 153–160. find
  51. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 1650–1659.
  52. Kevin Reschke, Martin Jankowiak, Mihai Surdeanu, Christopher D. Manning, and Daniel Jurafsky. 2014. Event Extraction Using Distant Supervision.. In Proceedings of the International Conference on Language Resources and Evaluation. 4527–4531. find
  53. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of EMNLP. Edinburgh, Scotland, 1524–1534.
  54. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M. Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluations.
  55. Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From Argumentation Mining to Stance Classification. In Proceedings of the Workshop on Argumentation Mining. Denver, Colorado, USA, 67–77. find
  56. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop CAAGET. 116–124.
  57. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 1555–1565.
  58. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 327–335.
  59. Andranik Tumasjan, Timm O. Sprenger, Philipp G. Sandner, and Isabell M. Welpe. 2010. Election Forecasts With Twitter: How 140 Characters Reflect the Political Landscape. Social Science Computer Review 29, 4 (2010), 402–418. find
  60. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 384–394.
  61. Peter D. Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In Proceedings of the Association for Computational Linguistics. 417–424.
  62. Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012a. Stance classification using dialogic properties of persuasion. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 592–596.
  63. Marilyn A. Walker, Jean E. Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012b. A Corpus for Research on Deliberation and Debate. In Proceedings of the International Conference on Language Resources and Evaluation. 812–817.
  64. Theresa Wilson, Zornitsa Kozareva, Preslav Nakov, Sara Rosenthal, Veselin Stoyanov, and Alan Ritter. 2013. SemEval-2013 Task 2: Sentiment Analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluation. Atlanta, USA.
  65. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Vancouver, British Columbia, Canada, 347–354.
  66. Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. 152–158.
  67. Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection. In Proceedings of the International Workshop on Semantic Evaluation. San Diego, California.
  68. Received January 2016; revised ..; accepted .. find
  69. ACM Transactions on Embedded Computing Systems, Vol. 0, No. 0, Article 0, Publication date: 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2016-stance-sentiment-tweets/