Elamur Research
☰
home / papers / 2017 / Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-1169.pdf

Summary

Tasks such as sentiment, emotion and sarcasm detection have little labelled data. Earlier distant-supervision work used narrow signals, such as binary positive/negative emoticons or hashtags that researchers mapped to emotion categories by hand. Felbo et al. use emojis instead, which give a much richer set of noisy labels. They collected 1246 million English tweets from 2013–2017 that contain no URLs and include at least one of 64 common emojis. After balancing the classes, they trained DeepMoji to predict which emoji appeared with a text. The model has a 256-dimensional embedding layer, two BiLSTM layers with 1024 units each, and an attention layer with skip-connections. To adapt the model to new tasks, they propose "chain-thaw", which fine-tunes one layer at a time before tuning the whole network. A single pretrained model set new state-of-the-art results on 8 benchmarks across 5 domains. Further analysis shows that the variety of emojis drives much of this gain. The released model is a reusable starting point for affect analysis when annotations are scarce.

Key points

  • Pretraining data: 1246M tweets labelled with 64 emoji types. A tweet with several distinct emojis is stored once for each emoji type, so pretraining stays a single-label task. The training set is balanced by upsampling.
  • Benchmarks: SE0714, Olympic and PsychExp (emotion); SS-Twitter, SS-Youtube and SE1604 (sentiment); SCv1 and SCv2-GEN (sarcasm).
  • Why pretraining helps: It increases word coverage on test sets (for example, SE0714 rises from 41.9% to 94.0%). Phrase-level context also matters: used as a feature extractor on SS-Youtube, an emoji-pretrained fastText model scores 63%, while DeepMoji scores 93%.
  • Comparison with humans: On a new set of 7,347 tweets, each rated by at least 10 MTurk workers, DeepMoji agrees with the crowd consensus 82.4% of the time. A single random rater agrees 76.1% of the time (fastText 71.0%, chance 50.1%).
  • Released resources: the pretrained model and code at github.com/bfelbo/DeepMoji (Theano original plus a Keras version), and an online demo at deepmoji.mit.edu.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

186 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 174 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1169 · arXiv 1708.00524

@inproceedings{felbo2017using,
  title = {{Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm}},
  author = {Felbo, Bjarke and Mislove, Alan and Søgaard, Anders and Rahwan, Iyad and Lehmann, Sune},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1169},
  eprint = {1708.00524},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1169}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR). find our card
  2. Yoshua Bengio et al. 2012. Deep learning of representations for unsupervised and transfer learning. In 29th International Conference on Machine learning (ICML) – Workshop on Unsupervised and Transfer Learning, volume 27, pages 17–36. find
  3. Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem - dimensional models and their implications on emotion representation and metrical evaluation. In 22nd European Conference on Artificial Intelligence (ECAI). 4 Available with preprocessing code, examples of usage, benchmark datasets etc. at github.com/bfelbo/deepmoji François Chollet et al. 2015. Keras. https:// github.com/fchollet/keras. ↗
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In 25th International Conference on Machine learning (ICML), pages 160–167. find
  5. Jan Deriu, Maurice Gonzenbach, Fatih Uzdilli, Aurelien Lucchi, Valeria De Luca, and Martin Jaggi. 2016. Swisscheese at semeval-2016 task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision. Proceedings of SemEval, pages 1124–1128. find
  6. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. Decaf: A deep convolutional activation feature for generic visual recognition. In 31th International Conference on Machine Learning (ICML), volume 32, pages 647–655. find
  7. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In 4th International Workshop on Natural Language Processing for Social Media (So-cialNLP). find
  8. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research (JMLR), 11:625–660. find
  9. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In 30th Conference on Neural Information Processing Systems (NIPS), pages 1019– 1027. find our card
  10. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, 1(12). find
  11. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. ↗
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  13. Xia Hu, Jiliang Tang, Huiji Gao, and Huan Liu. 2013. Unsupervised sentiment analysis with emotional signals. In Proceedings of the 22nd international conference on World Wide Web (WWW), pages 607–618. ACM. find
  14. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  15. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. ↗
  16. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR). find
  17. FA Kunneman, CC Liebrecht, and APJ van den Bosch. 2014. The (un)predictability of emotional hashtags in twitter. In 52th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics. find
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In 27th Conference on Neural Information Processing Systems (NIPS), pages 3111–3119. find
  19. Saif Mohammad. 2012. #emotional tweets. In The First Joint Conference on Lexical and Computational Semantics (*SEM), pages 246–255. Association for Computational Linguistics. find
  20. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. Semeval-2016 task 4: Sentiment analysis in twitter. In 10th International Workshop on Semantic Evaluation (SemEval), pages 1–18. find
  21. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing a diverse corpus of sarcasm in dialogue. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), page 31. find our card
  22. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. ↗ our card
  23. Jonathon Read. 2005. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In ACL student research workshop, pages 43–48. Association for Computational Linguistics. find
  24. Hassan Saif, Miriam Fernandez, Yulan He, and Harith Alani. 2013. Evaluation datasets for twitter sentiment analysis: a survey and a new dataset, the sts-gold. In Workshop: Emotion and Sentiment in Social and Expressive Media: approaches and perspectives from AI (ESSEM) at AI*IA Conference. find
  25. Valentina Sintsova, Claudiu-Cristian Musat, and Pearl Pu. 2013. Fine-grained emotion recognition in olympic tweets based on human computation. In 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA). find
  26. Jonas Sjöberg and Lennart Ljung. 1995. Overtraining, regularization and searching for a minimum, with application to neural networks. International Journal of Control, 62(6):1391–1407. find
  27. Jacopo Staiano and Marco Guerini. 2014. De-pechemood: A lexicon for emotion analysis from crowd-annotated news. In 52th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics. find
  28. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J Smola, and Eduard H Hovy. 2016. Hierarchical attention networks for document classification. In HLT-NAACL. find our card
  29. Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In 4th International Workshop on Semantic Evaluations (SemEval), pages 70–74. Association for Computational Linguistics. find
  30. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In 28th Conference on Neural Information Processing Systems (NIPS), pages 3104–3112. find
  31. Jared Suttles and Nancy Ide. 2013. Distant supervision for emotion classification with discrete binary values. In International Conference on Intelligent Text Processing and Computational Linguistics (CI-CLing), pages 121–136. Springer. find
  32. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In 52th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1555–1565. find
  33. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688. ↗ our card
  34. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology (JASIST), 63(1):163–173. find
  35. Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12):2544–2558. find
  36. Marilyn A Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In International Conference on Language Resources and Evaluation (LREC), pages 812–817. find
  37. Harald G Wallbott and Klaus R Scherer. 1986. How universal and specific is emotional experience? evidence from 27 countries on five continents. International Social Science Council, 25(4):763–795. find
  38. Michael Wiegand, Alexandra Balahur, Benjamin Roth, Dietrich Klakow, and Andrés Montoyo. 2010. A survey on the role of negation in sentiment analysis. In Workshop on Negation and Speculation in Natural Language Processing (NeSp-NLP), pages 60–68. Association for Computational Linguistics. find