Elamur Research⌕
☰
home / papers / 2017 / Textmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets

Textmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets

Paper · 2017 · Hardik Meisheri, Rupsa Saha, Priyanka Sinha et al.

Summary

This paper presents the Textmining team's system for the WASSA-2017 Emotion Intensity (EmoInt) shared task. The task is to predict a real-valued score between 0 and 1 that says how strongly a tweet expresses a given emotion: anger, fear, joy or sadness. The authors clean the tweets and turn them into pre-trained word embeddings: GloVe Twitter vectors for three emotions and Edinburgh embeddings for sadness. Two parallel neural branches read these embeddings, a CNN that captures n-gram cues and an LSTM that captures word order. Their outputs are joined with two hand-crafted feature groups: 45 lexicon-based features and one SentiNeuron sentiment signal. A small dense network then regresses the intensity. A separate model is trained for each emotion, and a correlation-based loss is used for fear and sadness. The submission placed 8th. Lexicon features gave a clear gain on the development data, but the model ranking did not carry over to the test set, where a plain LSTM beat the combined model. The work shows how far feature-augmented neural regression can go on short, noisy text.

Key points

  • Data: English tweets with one file per emotion (anger, fear, joy, sadness) for each of train, dev and test. Each tweet carries an intensity score in [0,1], and each emotion gets its own model.
  • Feature Set A: 45 features computed with the official baseline procedure. Sources include MPQA, Bing Liu, AFINN, Sentiment140, several NRC lexicons (Hashtag Sentiment, Hashtag Emotion, Word-Emotion, NRC-10 Expanded) and SentiWordNet, plus emoticon and negation features.
  • Feature Set B: the activation of unit 2388 in the SentiNeuron model, a 4096-unit LSTM language model trained on about 82M Amazon reviews, rescaled to the range 0–1.
  • Architecture:
    • The CNN branch has three parallel layers (bigram, trigram and 4-gram windows) with two filters each and max pooling, giving 6 values.
    • The LSTM branch has 10 units and gives 10 values.
    • Both outputs are merged with the 46 features, passed through a dense layer of 10 units, and end in a single output.
    • Tweets are capped at 35 tokens.
  • Results:
    • The unified model reached an average dev Pearson of 0.661.
    • In the feature ablation, using no features gave 0.514, Set B alone 0.571, Set A alone 0.655, and A+B 0.661.
    • The 1 − Pearson loss improved fear (0.690 vs 0.636) and sadness (0.649 vs 0.556) on dev compared with MSE.

Interesting ideas

  • I1. Instead of training only on squared error, the authors trained some models directly on one minus the Pearson correlation, the same quantity used for evaluation, and kept whichever loss worked better per emotion on dev.
  • I2. A feature group that hurt performance on its own (SentiNeuron) still gave a small gain once paired with the lexicon features, which a single all-or-nothing test would have hidden.
  • I3. The configuration that ranked best on dev lost to a simpler LSTM on test, which the authors attribute to vocabulary mismatch between the splits.
  • I4. Emoticons were removed from the embedding input but kept for lexicon feature extraction, so each signal was routed to the component that could actually use it.
  • I5. Readable lexicon scores (polarity, emotion association, negation counts) sat alongside learned representations and carried most of the measurable gain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Scoring heads (e.g., confidence or intensity estimators) could be trained against the ranking or correlation metric they are judged by, with the loss chosen per subtask on held-out data.
  • I2 (learn): Evaluation reports could ablate each knowledge source alone and in combination, so a weak-looking source is not dropped before its interaction effect is checked.
  • I3 (learn): Model selection should be checked on a second, differently sourced split, with vocabulary-overlap statistics recorded as a control before declaring a winner.
  • I4 (tokens): The tokenizer could emit emoticons and hashtags as distinct typed tokens (strip the "#" but mark the word as a hashtag), keeping them out of the syntactic stream while preserving them as cues.
  • I5 (frames): Lexicon-derived emotion and intensity values could be attached to frame participants and predicates as explicit, traceable attributes, supporting irony or speech-act judgements with inspectable evidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Hardik Meisheri, Rupsa Saha, Priyanka Sinha and Lipika Dey. 2017. Textmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5226

@inproceedings{meisheri2017textmining,
  title = {{Textmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets}},
  author = {Hardik Meisheri, Rupsa Saha, Priyanka Sinha and Lipika Dey},
  booktitle = {Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2017},
  doi = {10.18653/v1/w17-5226},
  url = {https://doi.org/10.18653/v1/w17-5226}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Felipe Bravo-Marquez, Eibe Frank, Saif M Mohammad, and Bernhard Pfahringer. 2016. Determining word–emotion associations from tweets by multi-label classification. In WI’16. IEEE Computer Society, pages 536–539. find
  2. Saif M. Mohammad and Felipe Bravo-Marquez. 2017. Wassa-2017 shared task on emotion intensity. In Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), September 2017, Copenhagen, Denmark.
  3. Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the 2008 international conference on web search and data mining. ACM, pages 231– 240. find
  4. Saif M Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence 31(2):301–326. ↗
  5. Andrea Esuli and Fabrizio Sebastiani. 2007. Sentiwordnet: A high-coverage lexical resource for opinion mining. Evaluation pages 1–26. find
  6. Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242 . ↗
  7. Yoav Goldberg. 2016. A primer on neural network models for natural language processing. Journal of Artificial Intelligence Research 57:345–420. ↗
  8. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29(3):436–465. ↗
  9. T Hercig, T Brychcın, L Svoboda, and M Konkol. 2016. Uwb at semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), Association for Computational Linguistics, San Diego, California. pages 354–361.
  10. Finn Årup Nielsen. 2011. A new anew: Evaluation of a word list for sentiment analysis in microblogs. arXiv preprint arXiv:1103.2903 . ↗
  11. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532– 1543. ↗
  12. Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent . find
  13. Sasa Petrovic, Miles Osborne, and Victor Lavrenko. 2010. The edinburgh twitter corpus. In Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media. pages 25–26. find
  14. Amal Htait, Sebastien Fournier, and Patrice Bellot. 2016. Lsis at semeval-2016 task 7: Using web search engines for english and arabic unsupervised sentiment intensity prediction. Proceedings of SemEval pages 469–473.
  15. Soujanya Poria, Erik Cambria, and Alexander F Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In EMNLP. pages 2539–2544. ↗
  16. Rie Johnson and Tong Zhang. 2014. Effective use of word order for text categorization with convolutional neural networks. arXiv preprint arXiv:1412.1058 . ↗
  17. A. Radford, R. Jozefowicz, and I. Sutskever. 2017. Learning to Generate Reviews and Discovering Sentiment. ArXiv e-prints . ↗
  18. Eshrag Refaee and Verena Rieser. 2016. ilab-edinburgh at semeval-2016 task 7: A hybrid approach for determining sentiment intensity of arabic twitter phrases. Proceedings of SemEval pages 474– 480.
  19. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. Semeval-2015 task 10: Sentiment analysis in twitter. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015). pages 451–463.
  20. Feixiang Wang, Zhihua Zhang, and Man Lan. 2016. Ecnu at semeval-2016 task 7: An enhanced supervised learning method for lexicon sentiment intensity ranking. Proceedings of SemEval pages 491– 496. find
  21. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, pages 347–354. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meisheri-2017-textmining-emoint-2017-deep-learning/