Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 6: Detecting Stance in Tweets

SemEval-2016 Task 6: Detecting Stance in Tweets

Summary

This paper describes the first shared task on stance detection in tweets. Given a tweet and a pre-chosen target, a system must decide whether the author is in favor, against, or neither. The organizers built a dataset of 4,870 English tweets covering six targets well known in the US. Tweets were collected through stance-indicative hashtags, which were then removed so they could not act as trivial cues. Crowd workers on CrowdFlower labeled each tweet.

There were two subtasks. Task A was standard supervised classification on five targets. Task B was zero-shot: it used a held-out target, "Donald Trump", with no labeled training data. Nineteen teams entered Task A and nine entered Task B, with top scores of 67.82 and 56.28. No Task A system outperformed a plain SVM baseline using word and character n-grams. All systems struggled when the tweet's opinion was aimed at a different entity than the target.

The work matters because it separates stance from sentiment, provides a reusable benchmark, and shows that cross-target generalization is still a hard problem.

Key points

  • Dataset: 4,870 tweets across six targets.
    • Task A covered five of them: Hillary Clinton, Legalization of Abortion, Feminist Movement, Atheism, and Climate Change is a Real Concern.
    • Task A split: about 70% training and 30% test per target.
    • Task B: all tweets for Donald Trump, used only as test data.
  • Collection and filtering:
    • More than two million tweets were gathered via query hashtags.
    • Retweets and tweets containing URLs were dropped.
    • Only tweets whose query hashtag appeared at the end were kept, and that hashtag was then stripped.
    • Up to 1,000 tweets per target were sampled evenly from favor, against, and stance-ambiguous hashtags.
  • Annotation:
    • Each pair was labeled by at least eight CrowdFlower annotators.
    • Raw inter-annotator agreement was 73.1%.
    • Only items where at least 60% of annotators agreed were retained.
    • Explicit "neutral" labels made up under 1%, so they were merged with "no clue" into a single "neither" class.
  • Task B results:
    • The winner, pkudblab (Favg 56.28), trained a deep CNN on a domain corpus labeled by keyword rules, then combined its output with rules to predict "neither".
    • This score exceeded the best supervised result on the Climate Change target (54.86).
  • Baselines: An SVM trained on all Task A targets combined could not beat the Majority baseline on the Trump test set.

Interesting ideas

  • I1. Strip the collection signal. The hashtags used to gather tweets were removed, and only tweets where they appeared at the end were kept, so the remaining text stays readable. This lets noisy weak labels be used for sampling without leaking the answer, which suits any dataset collected via keywords or emoji.
  • I2. Balanced sampling across query types. Sampling equally from favor, against, and ambiguous hashtags guarantees enough examples of every class before costly manual annotation begins. This is useful whenever natural class frequencies are skewed.
  • I3. Fold rare classes into a broader one. When the "neutral" class turned out to be almost empty, it was merged with "no inferable stance". Defining categories around what can actually be inferred, rather than what is theoretically possible, keeps the label scheme realistic.
  • I4. Report results by subset. Splitting test results by whether the opinion targets the given entity or a different one exposed a weakness that the overall score hid. Diagnostic breakdowns like this are worth adding to any classification benchmark.
  • I5. Bootstrap a new target with rules. For a target without labels, the top Task B teams generated noisy training labels from keyword or hashtag rules and then trained a model on them. This is a cheap recipe for low-resource settings, such as new topics or under-resourced languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

352 papers of our library cite it (of the 140884 whose bibliographies we read); 64 of them have a page here, 288 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1003

@inproceedings{mohammad2016semeval,
  title = {{SemEval-2016 Task 6: Detecting Stance in Tweets}},
  author = {Mohammad, Saif and Kiritchenko, Svetlana and Sobhani, Parinaz and Zhu, Xiaodan and Cherry, Colin},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1003},
  url = {https://doi.org/10.18653/v1/s16-1003}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis, pages 1–9. find
  2. Isabelle Augenstein, Andreas Vlachos, and Kalina Bontcheva. 2016. USFD at SemEval-2016 Task 6: Any-Target Stance Detection on Twitter with Autoencoders. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. Filip Boltuzic and Jan Šnajder. 201… find
  3. Alexander Conrad, Janyce Wiebe, and Rebecca Hwa. 2012. Recognizing arguing subjectivity and argument tags. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 80–88. find
  4. Marcelo Dias and Karin Becker. 2016. INF-UFRGS-OPINION-MINING at SemEval-2016 Task 6: Automatic Generation of a Training Corpus for Unsupervised Identification of Stance in Tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  5. Sarah Djemili, Julien Longhi, Claudia Marinica, Dimitris Kotzinos, and Georges-Elia Sarfati. 2014. What does Twitter have to say about ideology? In Proceedings of the Natural Language Processing for Computer-Mediated Communication/Social Media-Pre-conference workshop at Konvens. find
  6. Heba Elfardy and Mona Diab. 2016. CU-GWU at SemEval-2016 Task 6: Perspective at SemEval-2016 Task 6: Ideological Stance Detection in Informal Text. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  7. Adam Faulkner. 2014. Automated classification of stance in student essays: An approach using stance target information and the Wikipedia link-based measure. In Proceedings of the Twenty-Seventh International Flairs Conference. find
  8. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. find
  9. Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. 2014a. NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’14, Dublin, Ireland, August. find our card
  10. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014b. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. find
  11. Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In Charu C. Aggarwal and ChengXiang Zhai, editors, Mining Text Data, pages 415–463. find
  12. Springer US. Amita Misra, Brian Ecker, Theodore Handleman, Nicolas Hahn, and Marilyn Walker. 2016. nldsucsc at SemEval-2016 Task 6: A Semi-Supervised Approach to Detecting Stance in Tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  13. Saif M Mohammad and Peter D Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 26–34. find our card
  14. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June. find our card
  15. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of 10th edition of the the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  16. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2016b. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, Submitted. find
  17. Saif M Mohammad. 2015. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. find
  18. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2):1–135. find
  19. Braja Gopal Patra, Dipankar Das, and Sivaji Bandyopadhyay. 2016. JU NLP at SemEval-2016 Task 6: Detecting Stance in Tweets using Support Vector Machines. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  20. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’14, Dublin, Ireland, August. find our card
  21. Maria Pontiki, Dimitrios Galanis, Haris Papageogiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 Task 12: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’15, Denver, Colorado. find our card
  22. Ashwin Rajadesingan and Huan Liu. 2014. Identifying users with opposing opinions in Twitter debates. In Proceedings of the Conference on Social Computing, Behavioral-Cultural Modeling and Prediction, pages 153–160, Washington, DC, USA. find
  23. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. Semeval-2015 Task 10: Sentiment analysis in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluations. find our card
  24. Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From argumentation mining to stance classification. In Proceedings of the Workshop on Argumentation Mining, pages 67–77, Denver, Colorado, USA. find
  25. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. find our card
  26. Dhanya Sridhar, Lise Getoor, and Marilyn Walker. 2014. Collective stance classification of posts in online debate forums. Proceedings of the Association for Computational Linguistics, page 109. Martin Tutek, Ivan Sekulić, Paula Gombar, Ivan Paljak, Filip Čulinović, Filip Boltužić, Mladen Karan,… find
  27. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at SemEval-2016 Task 6: A Specific Convolutional Neural Network System for Effective Stance Detection. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find our card
  28. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347–354, Vancouver, British Columbia, Canada. find our card
  29. Theresa Wilson, Zornitsa Kozareva, Preslav Nakov, Sara Rosenthal, Veselin Stoyanov, and Alan Ritter. 2013. SemEval-2013 Task 2: Sentiment analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June. find our card
  30. Michael Wojatzki and Torsten Zesch. 2016. ltl.uni-due at SemEval-2016 Task 6: Stance Detection in Social Media Using Stacked Classifiers. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  31. Igarashi Yuki, Komatsu Hiroya, Kobayashi Sosuke, Okazaki Naoaki, and Inui Kentaro. 2016. Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, Californi… find
  32. Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/