Elamur Research⌕
☰
home / papers / 2017 / WASSA-2017 Shared Task on Emotion Intensity

WASSA-2017 Shared Task on Emotion Intensity

Summary

The paper describes the first shared task on estimating how strongly the author of a tweet experiences a given emotion. Given a tweet and one of anger, fear, joy, or sadness, a system outputs a real-valued intensity between 0 and 1. The organizers built the Tweet Emotion Intensity Dataset by collecting tweets that contained emotion-related query words taken from Roget's Thesaurus categories. Crowd annotators labeled the tweets with Best–Worst Scaling (BWS) rather than numeric rating scales, which yielded reliable fine-grained rankings.

The data was divided into training, development, and test splits and hosted on CodaLab. Submissions were scored by Pearson correlation with the gold intensities. Twenty-two teams took part, and the winning system, Prayas, reached 0.747. Seven teams beat the released SVM baseline, which scored 0.66.

Across participants, the strongest approaches combined word and sentence embeddings with features from many affect lexicons, usually inside model ensembles. The work treats basic emotions as graded rather than present-or-absent, and the resource fed into SemEval-2018 Task 1.

Key points

  • The task covers four emotions: anger, fear, joy, and sadness.
    • For each emotion, the organizers chose 50–100 query terms from Roget's Thesaurus categories (for example, category 900 "resentment" for anger).
    • Tweets were pulled from the Twitter API for three weeks starting November 22, 2016. Retweets and tweets containing URLs were dropped.
  • Annotation used BWS on 4-tuples.
    • Picking the best and worst items in one 4-tuple settles the relative ranking for five out of its six possible pairs.
    • About 2N tuples are enough to obtain reliable scores for N items.
  • Results:
    • 22 teams participated.
    • Prayas ranked first with Pearson r = 0.747.
    • The AffectiveTweets/Weka-based SVM baseline scored 0.66, and seven teams surpassed it.
  • Top systems:
    • Prayas used an ensemble of feed-forward, multi-task, and LSTM/CNN networks.
    • IMS (2nd) applied random forest regression over lexicon features, expanded-lexicon features, and neural network outputs.
    • SeerNet (3rd) ensembled SVR, AdaBoost, random forest, and gradient boosting, using GloVe, Word2Vec, and Emoji embeddings.
  • The eight best systems drew on noticeably more affect lexicons than weaker systems. Keras and TensorFlow were among the most widely used tools.

Interesting ideas

  • I1. Comparative annotation instead of direct scoring. Annotators pick only the most and least intense item in each group of four, and the choices are then converted into continuous scores. This sidesteps inconsistent ratings and scale-region bias, so it could help any project that needs graded labels, such as sentiment strength, toxicity, or text difficulty.
  • I2. Thesaurus-seeded data collection. The organizers chose query words spanning several intensity levels of one emotion, not just its obvious keywords. This makes the collected data cover the whole intensity range rather than clustering at the extremes.
  • I3. Multi-task sharing across related targets. Prayas shared the lower network layers across all four emotions and kept the upper layers separate for each one. This is a cheap way to transfer signal between related low-resource labels.
  • I4. Ensemble weights set by cross-validated quality. Each model's vote was weighted by how well it performed in cross-validation. In the paper, this improved on every individual model by at least two points, and it is a simple, general recipe for combining heterogeneous models.
  • I5. Lexicons still add value next to embeddings. Even deep-learning systems with large amounts of unlabeled data gained from features drawn from affect lexicons. Hand-built or expanded lexicons therefore remain worth keeping as complementary inputs, especially for languages or domains with little labeled data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5205

@inproceedings{mohammad2017wassa,
  title = {{WASSA-2017 Shared Task on Emotion Intensity}},
  author = {Mohammad, Saif and Bravo-Marquez, Felipe},
  booktitle = {Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2017},
  doi = {10.18653/v1/w17-5205},
  url = {https://doi.org/10.18653/v1/w17-5205}
}

References

47 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Esuli and Fabrizio Sebastiani. 2006. SENTI-WORDNET: A publicly available lexical resource for opinion mining. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC). Genoa, Italy, pages 417–422. find
  2. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Joint Conference on HLT–EMNLP. Vancouver, Canada. find our card
  3. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, Edward Elgar Publishing, pages 178–201. find
  4. Saima Aman and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue, volume 4629 of Lecture Notes in Computer Science, pages 196–205. find
  5. Nico H Frijda. 1988. The laws of emotion. American psychologist 43(5):349. find
  6. Johan Bollen, Huina Mao, and Alberto Pepe. 2009. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In Proceedings of the Fifth International Conference on Weblogs and Social Media. pages 450–453. find
  7. Kevin Gimpel, Nathan Schneider, et al. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). Portland, OR, USA. find our card
  8. Felipe Bravo-Marquez, Eibe Frank, Saif M. Mohammad, and Bernhard Pfahringer. 2016. Determining word–emotion associations from tweets by multi-label classification. In Proceedings of the 2016 IEEE/WIC/ACM International Conference on Web Intelligence. Omaha, NE, USA, pages 536–539. find
  9. Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van de Walle. 2015. Named entity recognition for twitter microposts using distributed word representations. ACL-IJCNLP 2015:146–153. find
  10. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explor. Newsl. 11(1):10–18. https://doi.org/10.1145/1656274.1656278. — [doi:10.1145/1656274.1656278](https://doi.org/10.1145/1656274.1656278) ↗
  11. Michael Brooks, Katie Kuksenok, Megan K Torkild-son, Daniel Perry, John J Robinson, Taylor J Scott, Ona Anicello, Ariana Zukowski, and Harris. 2013. Statistical affect detection in collaborative chat. In Proceedings of the 2013 conference on Computer supported cooperative work. San Antonio, Texas, U… find
  12. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, New York, NY, USA, pages 168–177. find
  13. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods 46(3):904–911. find
  14. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics. Atlanta, GA, USA. find
  15. LJ Cronbach. 1946. A case study of the splithalf reliability coefficient. Journal of educational psychology 37(8):473. find
  16. Herbert Aron David. 1963. The method of paired comparisons. Hafner Publishing Company, New York. find
  17. David Jurgens, Saif M. Mohammad, Peter Turney, and Keith Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation. Montréal, Canada, pages 356–364. find
  18. Peter Sheridan Dodds, Kameron Decker Harris, Isabel M. Kloumann, Catherine A. Bliss, and Christopher M. Danforth. 2011. Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. PloS One 6(12):e26752. find
  19. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (N… find
  20. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Int… find
  21. Svetlana Kiritchenko and Saif M. Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL). Vancouver, Canada. find our card
  22. Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2009. Compositionality principle in recognition of fine-grained emotions from text. In Proceedings of the Proceedings of the Third International Conference on Weblogs and Social Media (ICWSM-09). San Jose, California, pages 278–281. find
  23. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research (JAIR) 50:723–762. find
  24. Finn Årup Nielsen. 2011. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC Workshop on ’Making Sense of Microposts’: Big things come in small packages. Heraklion, Crete, pages 93–98. find
  25. G Frederic Kuder and Marion W Richardson. 1937. The theory of the estimation of test reliability. Psychometrika 2(3):151–160. find
  26. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. find
  27. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software, Inc. find
  28. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  29. W Parrot. 2001. Emotions in Social Psychology. Psychology Press. find
  30. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at ICLR. find
  31. Saša Petrović, Miles Osborne, and Victor Lavrenko. 2010. The Edinburgh Twitter corpus. In Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media. Association for Computational Linguistics, Stroudsburg, PA, USA, pages 25–26. find
  32. Saif Mohammad. 2012a. \#Emotional tweets. In The First Joint Conference on Lexical and Computational Semantics (*SEM 2012). Montréal, Canada. find
  33. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience 1(3):3–33. find
  34. Saif M. Mohammad. 2012b. From once upon a time to happily ever after: Tracking emotions in mail and books. Decision Support Systems 53(4):730–741. find
  35. Saif M. Mohammad. 2017. Word affect intensities. arXiv preprint arXiv:1704.08798 . find our card
  36. James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review 110(1):145. find
  37. Saif M. Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. In Proceedings of the sixth joint conference on lexical and computational semantics (*Sem). Vancouver, Canada. find our card
  38. Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In Proceedings of SemEval-2007. Prague, Czech Republic, pages 70– 74. find our card
  39. Saif M. Mohammad, Felipe Bravo-Marquez, Svetlana Kiritchenko, and Mohammad Salameh. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018). find our card
  40. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology 63(1):163–173. find
  41. Louis L. Thurstone. 1927. A law of comparative judgment. Psychological review 34(4):273. find
  42. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence 31(2):301–326. https://doi.org/10.1111/coin.12024. — [doi:10.1111/coin.12024](https://doi.org/10.1111/coin.12024) ↗
  43. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods 45(4):1191–1207. find
  44. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation. Atlanta, GA, USA. find our card
  45. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Joint Conference on HLT and EMNLP. Stroudsburg, PA, USA, pages 347–354. find our card
  46. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media 17(3). find
  47. Frank Yates. 1936. Incomplete randomized blocks. Annals of Human Genetics 7(2):121–140. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2017-wassa-2017-shared-task-emotion/