Elamur Research⌕
☰
home / papers / 2017 / Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus

Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus

Summary

Emotion research on social media has mostly used hashtags, emoticons or emojis as weak labels. No manually verified, publicly available Twitter corpus with discrete emotion categories existed. The authors close this gap by adding emotion labels to the SemEval 2016 Stance Data set, which already contains stance and sentiment annotation. Six trained student annotators labelled 4,868 tweets with any combination of Plutchik's eight emotions, and each tweet received three to six judgements. The release includes an aggregated version and every annotator's individual labels. The paper analyses inter-annotator agreement, the effect of different voting thresholds on the aggregated labels, and how emotions co-occur with stance and sentiment. It then trains linear (MaxEnt, SVM) and neural (LSTM, BiLSTM, CNN) classifiers as baselines. Neural models beat linear ones, and BiLSTM performs best. Unexpectedly, the most permissive aggregation, where a label counts if any annotator chose it, produces the strongest models. This suggests that the amount of labelled data matters more than consensus. The corpus supports joint research on emotion, sentiment and stance, and on individual differences in how people perceive emotion.

Key points

  • Corpus: 4,868 tweets taken from the 4,870 in the extended SemEval 2016 Stance Data set.
    • Labels: Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise and Trust, framed as tweet-level multi-label annotation.
    • Distribution: 696 tweets have 6 annotations, 703 have 5, 2,776 have 4 and 693 have 3.
    • Availability: data is published at ims.uni-stuttgart.de/data/ssec.
  • Agreement: pairwise Cohen's κ is highest for Anger (0.28–0.49), Joy (0.30–0.52) and Trust (0.29–0.57). It is low for Sadness, Disgust and Fear, with minimums of 0.04–0.08.
  • Label shrinkage: label counts drop sharply as the agreement threshold rises. For example, Anger falls from 2,902 labels at t=0.0 to 578 at t=0.99, and Surprise falls from 1,108 to 213.
  • Classifier results: MaxEnt and SVM reach about 58% micro-averaged F1. LSTM, BiLSTM and CNN improve on this by 3, 4 and 2 points respectively. BiLSTM is best on 5 of the 8 emotions.
  • Per-emotion performance: Anger is predicted best and Surprise worst. BiLSTM F1 declines steadily as the voting threshold increases, which motivates using t=0.0 as the canonical aggregation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5203

@inproceedings{schuff2017annotation,
  title = {{Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus}},
  author = {Schuff, Hendrik and Barnes, Jeremy and Mohme, Julian and Padó, Sebastian and Klinger, Roman},
  booktitle = {Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2017},
  doi = {10.18653/v1/w17-5203},
  url = {https://doi.org/10.18653/v1/w17-5203}
}

References

53 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saima Aman and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue: 10th International Conference, TSD 2007, Pilsen, Czech Republic, September 3-7, 2007. Proceedings, pages 196–205. Springer. find
  2. Isabelle Augenstein, Andreas Vlachos, and Kalina Bontcheva. 2016. USFD at semeval-2016 task 6: Any-target stance detection on twitter with autoencoders. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 389–393, San Diego, California. find
  3. Sarah Busch. 2011. Capturing mood about daily weather from twitter posts. http://www.dialogueearth.org/2011/09/29/capturing-mood-about-daily-weather-from-twitter-posts. ↗
  4. Sarah Busch. 2012. Tracking the mood about gas prices on twitter: A case study. http://www.dialogueearth.org/2012/01/25/tracking-the-mood-about-gas-prices-on-twitter-a-case-study. ↗
  5. Kent Cavender-Bares. 2011. Preparing to extract weather mood from tweets. http://www.dialogueearth.org/2011/03/03/preparing-to-extract-weather-mood-from-tweets. ↗
  6. STS-Gold. In Proceedings of the First International Workshop on Emotion and Sentiment in Social and Expressive Media: approaches and perspectives from AI (ESSEM 2013), pages 9–21, Turin, Italy. find
  7. Domenic V. Cicchetti and Alvan R. Feinstein. 1990. High agreement but low kappa: II. resolving the paradoxes. Journal of clinical epidemiology, 43:551–558. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  9. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD ’04 Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177, Seattle, WA, USA. find
  10. Joana Costa, Catarina Silva, Mario Antunes, and Bernardete Ribeiro. 2014. Concept drift awareness in twitter streams. In 13th International Conference on Machine Learning and Applications, pages 294– 299. find
  11. Jason S. Kessler, Miriam Eckert, Lyndsie Clark, and Nicolas Nicolov. 2010. find
  12. The 2010 ICWSM JDPA Sentiment Corpus for the Automotive Domain. In Proc. of the 4th International AAAI Conference on Weblogs and Social Media Data Workshop Challenge (ICWSM-DWC). find
  13. Marcelo Dias and Karin Becker. 2016. INF-UFRGS-OPINION-MINING at SemEval-2016 task 6: Automatic generation of a training corpus for unsupervised identification of stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 378–383, San Diego, Cali… find
  14. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. find our card
  15. Xiaowen Ding, Bing Liu, and Philip S. Yu. 2008. A holistic lexicon-based approach to opinion mining. In WSDM ’08 Proceedings of the 2008 International Conference on Web Search and Data Mining, pages 213–239, Palo Alto, California, USA. find
  16. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations, San Diego, CA, USA. find
  17. Peter S. Dodds, Kameron D. Harris, Isabel M. Kloumann, Catherine A. Bliss, and Christopher M. Danforth. 2011. Temporal patterns of happiness and information in a global social network: Hedonomet-rics and twitter. PloS one, 6(12). find
  18. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. find our card
  19. Javid Ebrahimi, Dejing Dou, and Daniel Lowd. 2016. Weakly supervised tweet stance classification by relational bootstrapping. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1012–1017, Austin, Texas. find
  20. Roman Klinger. 2017. Does optical character recognition and caption generation improve emotion detection in microblog posts? In Natural Language Processing and Information Systems: 22nd International Conference on Applications of Natural Language to Information Systems, NLDB 2017, Liège, Belgium, J… find
  21. Paul Ekman. 1999. Basic emotions. In M Dalgleish, T; Power, editor, Handbook of Cognition and Emotion. John Wiley & Sons, Sussex, UK. find
  22. Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2029–2041, Berlin, Germany. find
  23. Himabindu Lakkaraju, Chiranjib Bhattacharyya, Indra-jit Bhattacharya, and Srujana Merugu. 2011. Exploiting coherence for the simultaneous discovery of latent facets and associated sentiments. In Proceedings of the 2011 SIAM International Conference on Data Mining, pages 498–509, Mesa, Arizona, USA. find
  24. Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In International Workshop on the Web and Databases (WebDB 2009), Providence, Rhode Island, USA. find
  25. Tomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations, Scottsdale, AZ, USA. find
  26. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. Technical report, Stanford. find
  27. Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of WASSA at EMNLP, Copenhagen, Denmark. find
  28. Yulan He Hassan Saif, Miriam Fernandez and Harith Alani. 2013. Evaluation Datasets for Twitter Sentiment Analysis: A survey and a new dataset, the Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting stance in tweets. In Proceeding… find
  29. Cicero dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 69–78, Dublin, Ireland. find
  30. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, 17(3). find
  31. Klaus Scherer and Harald Wallbott. 1997. The ISEAR questionnaire and codebook. Geneva Emotion Research Group. find
  32. David A. Shamma, Lyndon Kennedy, and Elizabeth F. Churchill. 2009. Tweet the debates: Understanding community annotation of uncollected sources. In Proceedings of the First SIGMM Workshop on Social Media, pages 3–10, Beijing, China. find
  33. Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4):480 – 499. find
  34. Michael Speriosu, Nikita Sudan, Sid Upadhyay, and Jason Baldridge. 2011. Twitter polarity classification with label propagation over lexical links and the follower graph. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 53–63, Edinburgh, Scotland. find
  35. Myriam Munezero, Calkin Suero Montero, Erkki Su-tinen, and John Pajunen. 2014. Are they different? Affect, feeling, emotion, sentiment, and opinion detection in text. IEEE Transactions on Affective Computing, 5(2):101–111. find
  36. Damiano Spina, Edgar Meij, Maarten de Rijke, Andrei Oghina, Minh Thuong Bui, and Mathias Breuss. 2012. Identifying entity aspects in microblog posts. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1089–1090, New York, NY, USA… find
  37. Preslav Nakov, Alan Ritter, Sara Rosenthal, Veselin Stoyanov, and Fabrizio Sebastiani. 2016. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California. Association for Computational Linguistics. find
  38. Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 Task 2: Sentiment analysis in twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Sema… find
  39. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929– 1958. find
  40. Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective Text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74, Prague, Czech Republic. find
  41. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin,… find our card
  42. Jared Suttles and Nancy Ide. 2013. Distant supervision for emotion classification with discrete binary values. In Computational Linguistics and Intelligent Text Processing: 14th International Conference, CI-CLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part II, pages 121–136, Berlin, He… find
  43. Robert Plutchik. 2001. The nature of emotions. American Scientist, 89(July–August):344–350. find
  44. Matthew Purver and Stuart Battersby. 2012. Experimenting with distant supervision for emotion classification. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–491, Avignon, France. find
  45. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
  46. Kirk Roberts, Michael A. Roach, Joseph Johnson, Josh Guthrie, and Sanda M. Harabagiu. 2012. Em-patweet: Annotating and detecting emotions on twitter. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 3806–3813, Istanbul, Turkey. find
  47. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective LSTMs for target-dependent sentiment classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3298– 3307, Osaka, Japan. find
  48. Mike Thelwall, Kevan Buckley, and Georgios Pal-toglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science Technology, 63(1):163–173. find
  49. Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575–584, Uppsala, Sweden. find
  50. Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P. Sheth. 2012. Harnessing twitter ”big data” for automatic emotion identification. In 2012 ASE/IEEE International Conference on Social Computing and 2012 ASE/IEEE International Conference on Privacy, Security, Risk and Trust, pages 587–592,… find
  51. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at SemEval-2016 Task 6: A specific convolutional neural network system for effective stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 384–388, San Diego, Califo… find our card
  52. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3). find
  53. Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485–… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/schuff-2017-annotation-modelling-analysis-fine-grained/