Elamur Research⌕
☰
home / papers / 2020 / Annotated Corpus of Tweets in English from Various Domains for Emotion Detection

Annotated Corpus of Tweets in English from Various Domains for Emotion Detection

Paper · 2020 · Soumitra Ghosh, Asif Ekbal, Pushpak Bhattacharyya et al.

Summary

Most emotion datasets cover a narrow set of sources, such as blogs, fairy tales, news headlines, weather or election tweets. Specialised areas like security, social issues, technology, politics and sports have little labelled data. The authors address this gap with an English Twitter corpus of 18,921 tweets. The tweets were gathered with keywords such as terror, cyber-security and technology between January 2018 and August 2019, then filtered with lexicons and cleaned.

Three annotators gave each tweet a single label: one of Ekman's six basic emotions or a non-emotive Others class. Agreement reached an average Cohen's kappa of 0.74.

On this benchmark the authors compare CNN, Bi-LSTM and Bi-GRU classifiers with HAtED, a Bi-GRU that applies attention first over words and then over sentences. HAtED performs best among the 7-class systems, with about 69% test accuracy and macro F1 of 0.46. Variants trained on 4 or 6 classes show how rare classes and the catch-all class affect results. The resource matters because it gives emotion detection a multi-domain social-media testbed.

Key points

  • The corpus has 18,921 English tweets collected through Twython from Twitter's Standard search API. Questions, requests, poems and code-mixed tweets were removed.
  • Labels are Anger, Disgust, Fear, Joy/Happiness, Sadness, Surprise and Others, with one label per tweet at document level.
  • Average Cohen's kappa is 0.74. Joy has the highest per-class agreement (0.87), then Others (0.83), and Surprise the lowest (0.62).
  • In the 7-class setting, macro F1 is 0.32 for CNN, 0.39 for Bi-GRU, 0.32 for Bi-LSTM and 0.46 for HAtED. HAtED reaches about 69% accuracy.
  • HAtED 4-C reaches macro F1 0.66 after dropping the rare Disgust and Surprise classes. HAtED 6-C, which drops Others, also beats the full 7-class model.

Interesting ideas

  • I1. The authors retrain the same model on reduced label sets (7, 6 and 4 classes) to measure how much the rare classes and the catch-all class each cost, much like the known effect of a Neutral class in sentiment tasks.
  • I2. The paper reports kappa per class rather than only an average, which exposes Surprise as unstable; the authors partly attribute this to the class mixing pleasant and unpleasant surprises.
  • I3. In the error analysis, the main failures are implicit tweets with no overt affect words, and words that appear mostly in negative contexts pulling positive tweets toward negative labels.
  • I4. Preprocessing replaces emoticons with word meanings (":-(" becomes sad, "<3" becomes love) and expands common contractions before sentence splitting.
  • I5. HAtED treats a tweet as a sequence of sentences and weighs both the words inside each sentence and the sentences themselves before deciding on a label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when evaluating speech-act or irony classifiers, report scores with and without the "other/none" bucket, so it is clear whether errors come from real distinctions or from the dump class.
  • I2 (frames): when designing pragmatic labels, check per-label agreement among witnesses and split labels whose valence is mixed, such as surprise, into polarity-specific subtypes.
  • I3 (frames): emotion should be inferred from the event frame (who suffered or gained what), not from cue words, and such tweets make natural negative controls for testing that.
  • I4 (tokens): map emoticons to lexical tokens with a lemma and POS, so they can attach in the UD tree and feed affect into frames rather than being discarded as noise.
  • I5 (world): record affective states per sentence as explicit states in the text world, then derive the document-level emotion from them in a traceable way.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology PatnaPatna · India21 works · 374 citations🎓Islamic Azad University, TehranTehran · Iran2 works · 5 citations

Also on the byline: Jaypee University of Engineering and Technology, Guna · Centre for Development of Telematics · GlaxoSmithKline (United Kingdom).

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Soumitra Ghosh, Asif Ekbal, Pushpak Bhattacharyya, Sriparna Saha, Vipin Tyagi, Alka Kumar, Shikha Srivastava and Nitish Kumar. 2020. Annotated Corpus of Tweets in English from Various Domains for Emotion Detection. Proceedings of the 17th International Conference on Natural Language Processing (ICON).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ghosh2020annotated,
  title = {{Annotated Corpus of Tweets in English from Various Domains for Emotion Detection}},
  author = {Soumitra Ghosh, Asif Ekbal, Pushpak Bhattacharyya, Sriparna Saha, Vipin Tyagi, Alka Kumar, Shikha Srivastava and Nitish Kumar},
  booktitle = {Proceedings of the 17th International Conference on Natural Language Processing (ICON)},
  year = {2020},
  url = {https://aclanthology.org/2020.icon-main.62/}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md Shad Akhtar, Ayush Kumar, Asif Ekbal, and Pushpak Bhattacharyya. 2016. A hybrid deep learning architecture for sentiment analysis. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 482–493. find
  2. Shad Akhtar, Deepanway Ghosal, Asif Ekbal, Pushpak Bhattacharyya, and Sadao Kurohashi. 2019. All-in-one: Emotion, sentiment and intensity prediction using a multi-task ensemble framework. IEEE Transactions on Affective Computing. ↗
  3. Ebba Cecilia Ovesdotter Alm. 2008. Affect in* text and speech. Citeseer. find
  4. Saima Aman and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In International Conference on Text, Speech and Dialogue, pages 196– 205. Springer. find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗
  6. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. ↗
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  8. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. ↗
  9. Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 152–165. Springer. find
  10. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315– 323. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  14. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957. ↗
  15. Navonil Majumder, Soujanya Poria, Haiyun Peng, Niyati Chhaya, Erik Cambria, and Alexander Gelbukh. 2019. Sentiment and sarcasm classification with multitask learning. IEEE Intelligent Systems, 34(3):38–43. ↗
  16. Saif M Mohammad and Felipe Bravo-Marquez. 2017. Wassa-2017 shared task on emotion intensity. arXiv preprint arXiv:1708.03700. ↗
  17. Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.
  18. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301–326. ↗
  19. Saif M Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3):26. ↗
  20. Saif M Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4):480–499. ↗
  21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  22. Robert Plutchik. 2001. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist, 89(4):344– 350. find
  23. Klaus R Scherer and Harald G Wallbott. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2):310. ↗
  24. Hendrik Schuff, Jeremy Barnes, Julian Mohme, Sebastian Padó, and Roman Klinger. 2017. Annotation, modelling and analysis of fine-grained emotions on a stance and sentiment detection corpus. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 13–23.
  25. Prerana Singhal and Pushpak Bhattacharyya. 2016. Borrow a little from your rich cousin: Using embeddings and polarities of english words for multilingual sentiment classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3053–3062. find
  26. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  27. Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74.
  28. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghosh-2020-annotated-corpus-tweets-english-various/