Elamur Research⌕
☰
home / papers / 2017 / Emotion Intensities in Tweets

Emotion Intensities in Tweets

Summary

Most emotion detection research assigns categorical labels, such as whether anger is present, rather than measuring how strongly an emotion is expressed. This paper addresses that gap by creating the first collection of tweets labelled with real-valued intensity scores (0 to 1) for anger, fear, joy and sadness. Tweets were gathered using emotion-related query terms taken from Roget's Thesaurus. They were then scored by crowd workers using Best–Worst Scaling (BWS): annotators see groups of four tweets and pick the most and least intense, and these choices are converted into fine-grained scores. The study shows that BWS works for whole sentences, not only single words. It also reveals that emotion-word hashtags usually make a tweet read as more intense. The authors release a Weka-based regression system, AffectiveTweets, and use it to compare feature types. Combining word embeddings with affect lexicon features gives the best result (average Pearson r = 0.66). The dataset became the basis of the WASSA-2017 Shared Task on Emotion Intensity, giving the community a standard benchmark for emotion regression.

Key points

  • Dataset: The Tweet Emotion Intensity dataset contains 7,097 tweets across four emotions. These include 1,030 tweets with a trailing hashtagged query term (HQT), 1,030 copies of those tweets with the hashtag removed (NQT), and 5,037 other query-term tweets (QT). The test portion has 3,142 tweets.
  • Query terms: Each emotion has 50–100 query terms drawn from Roget's categories: 900 Resentment, 860 Fear, 836 Cheerfulness and 837 Dejection. Collection was capped at 50 tweets per term and one tweet per tweeter–term pair. Retweets and tweets containing URLs were excluded.
  • Feature comparison (average Pearson r):
    • Word and character n-grams: about 0.48 each.
    • Word embeddings: 0.55.
    • Single lexicons: from 0.19 (SentiWordNet) to 0.53 (NRC-Hash-Emo).
    • All lexicons combined: 0.63.
    • Embeddings plus lexicons: 0.66 (best).
    • Significance was tested with a Wilcoxon signed-rank test over ten random data partitions.
  • Cross-emotion transfer:
    • A model trained on fear predicts sadness with r = 0.63, close to the 0.65 obtained by training on sadness itself.
    • Training on fear and sadness data together reaches 0.67 for sadness.
    • Joy correlates negatively with all three negative emotions.
  • Shared task and release: The data supported the WASSA-2017 shared task on CodaLab, in which 22 teams took part. Data, questionnaires, evaluation scripts and code are freely available.

Interesting ideas

  • I1. Relative judgements instead of rating scales. Asking annotators to pick the extremes within a 4-tuple yields five of the six pairwise orderings per judgement, and about 2N tuples suffice for reliable scores. This could cheaply produce graded labels for any subjective property, such as toxicity or formality in Ukrainian text.
  • I2. Paired hashtag-removed copies. Annotating each tweet alongside a copy without its trailing emotion hashtag isolates the effect of one explicit cue on perceived intensity. The same controlled-ablation design could test how much any single surface marker drives human judgements.
  • I3. Balanced, term-driven sampling. Seeding collection with thesaurus terms that span intensity levels, while capping tweets per term and per user, keeps a corpus rich in the target phenomenon without letting a few users or keywords dominate. This is useful for building any targeted social-media corpus.
  • I4. Cross-label transfer as a similarity measure. Training on one label and testing on another quantifies how alike two categories are in language, and shows that this similarity can be asymmetric. It also points to which auxiliary datasets could supplement scarce training data.
  • I5. Simple pooling of related data. Merging training sets of linguistically similar labels, here fear and sadness, improved results without any sophisticated adaptation. This is a cheap first step when data for a target label is limited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

70 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-1007

@inproceedings{mohammad2017emotion,
  title = {{Emotion Intensities in Tweets}},
  author = {Mohammad, Saif and Bravo-Marquez, Felipe},
  booktitle = {Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)},
  year = {2017},
  doi = {10.18653/v1/s17-1007},
  url = {https://doi.org/10.18653/v1/s17-1007}
}

References

51 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explor. Newsl. 11(1):10–18. https://doi.org/10.1145/1656274.1656278. — [doi:10.1145/1656274.1656278](https://doi.org/10.1145/1656274.1656278) ↗
  2. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Joint Conference on HLT–EMNLP. Vancouver, Canada. find
  3. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, New York, NY, USA, pages 168–177. find
  4. Saima Aman and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue, volume 4629 of Lecture Notes in Computer Science, pages 196–205. find
  5. Johan Bollen, Huina Mao, and Alberto Pepe. 2009. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In Proceedings of the Fifth International Conference on Weblogs and Social Media. pages 450–453. find
  6. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics. Atlanta, GA, USA. find
  7. Felipe Bravo-Marquez, Eibe Frank, Saif M Mohammad, and Bernhard Pfahringer. 2016. Determining word–emotion associations from tweets by multi-label classification. In Proceedings of the 2016 IEEE/WIC/ACM International Conference on Web Intelligence. Omaha, NE, USA, pages 536–539. find
  8. David Jurgens, Saif M. Mohammad, Peter Turney, and Keith Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation. Montréal, Canada, pages 356–364. find
  9. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (N… find
  10. Michael Brooks, Katie Kuksenok, Megan K Torkild-son, Daniel Perry, John J Robinson, Taylor J Scott, Ona Anicello, Ariana Zukowski, and Harris. 2013. Statistical affect detection in collaborative chat. In Proceedings of the 2013 conference on Computer supported cooperative work. San Antonio, Texas, U… find
  11. Svetlana Kiritchenko and Saif M. Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL). Vancouver, Canada. find
  12. LJ Cronbach. 1946. A case study of the splithalf reliability coefficient. Journal of educational psychology 37(8):473. find
  13. Herbert Aron David. 1963. The method of paired comparisons. Hafner Publishing Company, New York. find
  14. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research 50:723–762. find
  15. Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6(3):169–200. find our card
  16. Andrea Esuli and Fabrizio Sebastiani. 2006. SENTI-WORDNET: A publicly available lexical resource for opinion mining. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC). Genoa, Italy, pages 417–422. find
  17. G Frederic Kuder and Marion W Richardson. 1937. The theory of the estimation of test reliability. Psychometrika 2(3):151–160. find
  18. FA Kunneman, CC Liebrecht, and APJ van den Bosch. 2014. The (un) predictability of emotional hashtags in twitter. In Proceedings of the 5th Workshop on Language Analysis for Social Media. Gothenburg, Sweden, pages 26–34. find
  19. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, Edward Elgar Publishing, pages 178–201. find
  20. Nico H Frijda. 1988. The laws of emotion. American psychologist 43(5):349. find
  21. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. find
  22. Kevin Gimpel, Nathan Schneider, et al. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). Portland, OR, USA. find
  23. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  24. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at ICLR. find
  25. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1(12). find
  26. Saša Petrović, Miles Osborne, and Victor Lavrenko. 2010. The Edinburgh Twitter corpus. In Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media. Association for Computational Linguistics, Stroudsburg, PA, USA, pages 25–26. find
  27. Saif M. Mohammad. 2012. \#Emotional tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics. Montréal, Canada, SemEval ’12, pages 246–255. find
  28. Saif M Mohammad. 2017. Word affect intensities. arXiv preprint arXiv:1704.08798 . find our card
  29. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience 1(3):3–33. find
  30. Saif M. Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA). Copenhagen, Denmark. find
  31. Ashequl Qadir and Ellen Riloff. 2013. Bootstrapped learning of emotion hashtags\# hashtags4you. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis. Atlanta, GA, USA, pages 2–11. find
  32. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence 31(2):301–326. https://doi.org/10.1111/coin.12024. — [doi:10.1111/coin.12024](https://doi.org/10.1111/coin.12024) ↗
  33. Ashequl Qadir and Ellen Riloff. 2014. Learning emotion indicators from tweets: Hashtags, hashtag patterns, and phrases. In Proceedings of the EMNLP Workshop on Arabic Natural Langauge Processing (EMNLP). Doha, Qatar, pages 1203–1209. find
  34. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation. Atlanta, GA, USA. find
  35. Kirk Roberts, Michael A Roach, Joseph Johnson, Josh Guthrie, and Sanda M Harabagiu. 2012. Empatweet: Annotating and detecting emotions on Twitter. In Proceedings of the Conference on Language Resources and Evaluation. pages 3806–3813. find
  36. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media 17(3). find
  37. James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review 110(1):145. find
  38. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29(3):436–465. find
  39. Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In Proceedings of SemEval-2007. Prague, Czech Republic, pages 70– 74. find
  40. Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. July 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing and Management 51(4):480– 499. find
  41. Anja Summa, Bernd Resch, Geoinformatics-Z GIS, and Michael Strube. 2016. Microblog emotion classification by computing similarity in text, time, and space. In Proceedings of the PEOPLES Workshop at COLING. Osaka, Japan, pages 153–162. find
  42. Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2009. Compositionality principle in recognition of fine-grained emotions from text. In Proceedings of the Proceedings of the Third International Conference on Weblogs and Social Media (ICWSM-09). San Jose, California, pages 278–281. find
  43. Jared Suttles and Nancy Ide. 2013. Distant supervision for emotion classification with discrete binary values. In Computational Linguistics and Intelligent Text Processing, Springer, pages 121–136. find
  44. Finn Årup Nielsen. 2011. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC Workshop on ’Making Sense of Microposts’: Big things come in small packages. Heraklion, Crete, pages 93–98. find
  45. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology 63(1):163–173. find
  46. Louis L. Thurstone. 1927. A law of comparative judgment. Psychological review 34(4):273. find
  47. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. find
  48. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Joint Conference on HLT and EMNLP. Stroudsburg, PA, USA, pages 347–354. find
  49. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of the Conference on Language Resources and Evaluation (LREC). Malta. find our card
  50. Frank Yates. 1936. Incomplete randomized blocks. Annals of Human Genetics 7(2):121–140. find
  51. W Parrot. 2001. Emotions in Social Psychology. Psychology Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2017-emotion-intensities-tweets/