Elamur Research
☰
home / papers / 2018 / SemEval-2018 Task 1: Affect in Tweets

SemEval-2018 Task 1: Affect in Tweets

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/S18-1001.pdf

Summary

Earlier work on emotion in text mostly asks whether an emotion is present, and rarely how strong it is. This SemEval-2018 shared task asked systems to infer a tweeter's affective state through five subtasks. Two cover emotion intensity for anger, fear, joy, and sadness, as regression and as ordinal classification. Two do the same for valence. The fifth is multi-label classification over eleven emotions. The organizers built the AIT Dataset: roughly 22,000 English, Arabic, and Spanish tweets labelled with about 700,000 annotations. Intensity scores came from Best–Worst Scaling, a comparative scheme that avoids the inconsistency of rating scales, and repeat annotation confirmed the scores are stable. The competition drew 75 teams and 319 submissions. The strongest systems typically combined neural sentence embeddings with features from sentiment and emotion lexicons. A separate audit with the Equity Evaluation Corpus showed that most submitted systems scored one gender or race slightly but consistently higher. The freely released data and bias corpus offer a benchmark for fine-grained, multilingual affect modelling and for fairness testing.

Key points

  • Five subtasks across three languages give 15 task–language pairs: EI-reg, EI-oc (four ordinal classes), V-reg, V-oc (seven classes), and E-c (eleven emotions plus neutral).
  • Best–Worst Scaling on 4-tuples produced real-valued intensity scores with split-half reliability above 0.8. English data combines Tweets-2016 (from the EmoInt Dataset) with new Tweets-2017. Arabic and Spanish tweets come only from 2017.
  • About 200 people in 75 teams took part, which the authors describe as the largest SemEval task so far. Each team could make one official submission per task–language pair.
  • The bias audit covered 219 EI-reg and V-reg English submissions from 50 teams on template sentences from the EEC.
    • Only about 25% (emotion tasks) and 14% (valence) of submissions showed no significant gender difference.
    • For race, the share without a significant difference was 11–24% (emotion tasks) and about 8% (valence).
    • Score gaps were below 0.03 on the 0–1 scale.
  • Bias direction depended on the dimension. Female noun phrases got higher anger, joy, and valence scores, while male ones got higher fear scores. African American names received higher anger, fear, and sadness scores, while European American names received higher joy and valence scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

347 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 338 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1001

@inproceedings{mohammad2018semeval,
  title = {{SemEval-2018 Task 1: Affect in Tweets}},
  author = {Mohammad, Saif and Bravo-Marquez, Felipe and Salameh, Mohammad and Kiritchenko, Svetlana},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1001},
  url = {https://doi.org/10.18653/v1/s18-1001}
}

References

31 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (N… find
  2. Gilbert Badaro, Ramy Baly, Hazem M. Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing, pages 165– 173, Doha, Qatar. find
  3. Svetlana Kiritchenko and Saif M. Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. find
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS), pages 4349– 4357. find our card
  5. Svetlana Kiritchenko and Saif M. Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM). find our card
  6. Margaret M Bradley and Peter J Lang. 1999. Affective norms for English words (ANEW): Instruction manual and affective ratings. Technical report, The Center for Research in Psychophysiology, University of Florida. find
  7. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. find
  8. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153–163. find
  9. LJ Cronbach. 1946. A case study of the splithalf reliability coefficient. Journal of educational psychology, 37(8):473. find
  10. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  11. Fermı́n L Cruz, José A Troyan, Beatriz Pontes, and F Javier Ortega. 2014. Ml-senticon: Un lexicón multilingüe de polaridades semánticas a nivel de lemas. Procesamiento del Lenguaje Natural, (53). find
  12. G Frederic Kuder and Marion W Richardson. 1937. The theory of the estimation of test reliability. Psychometrika, 2(3):151–160. find
  13. Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion, 6(3):169–200. find our card
  14. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436. find
  15. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, Cope… find our card
  16. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. find
  17. Saif Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. Sentiment lexicons for arabic social media. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find
  18. T. N. Flynn and A. A. J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, pages 178–201. Edward Elgar Publishing. find
  19. Saif M. Mohammad. 2016. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Herb Meiselman, editor, Emotion Measurement. Elsevier. find
  20. Nico H Frijda. 1988. The laws of emotion. American psychologist, 43(5):349. find
  21. Sara Hajian and Josep Domingo-Ferrer. 2013. A methodology for direct and indirect discrimination prevention in data mining. IEEE Transactions on Knowledge and Data Engineering, 25(7):1445– 1459. find
  22. Saif M. Mohammad. 2018a. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of The Annual Meeting of the Association for Computational Linguistics (ACL), Melbourne, Australia. find our card
  23. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177, New York, NY, USA. ACM. find
  24. Saif M. Mohammad. 2018b. Word affect intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. find
  25. Grigori Sidorov, Sabino Miranda-Jiménez, Francisco Viveros-Jiménez, Alexander Gelbukh, Noé Castro-Sánchez, Francisco Velásquez, Ismael Dı́az-Rangel, Sergio Suárez-Guerra, Alejandro Treviño, and Juan Gordon. 2012. Empirical study of machine learning based approach for opinion mining in tweets.… find
  26. Saif M. Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Copenhagen, Denmark. find
  27. Saif M. Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. find
  28. Michael Wojatzki, Saif M. Mohammad, Torsten Zesch, and Svetlana Kiritchenko. 2018. Quantifying qualitative data for understanding controversial issues. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan. find
  29. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, 17(3). find
  30. Indre Zliobaite. 2015. A survey on measuring indirect discrimination in machine learning. arXiv preprint arXiv:1511.00148. — [arXiv:1511.00148](https://arxiv.org/abs/1511.00148) find
  31. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. find