Elamur Research⌕
☰
home / papers / 2017 / DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis

DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis

Summary

The paper describes two neural systems that the University of Piraeus team entered in SemEval-2017 Task 4 (Sentiment Analysis in Twitter). They covered every English subtask: message-level polarity (A), topic-based classification on 2- and 5-point scales (B, C), and quantification of per-topic sentiment distributions (D, E).

The authors avoid hand-built features, sentiment lexicons and ensembles. Instead they rely on three components:

  • Preprocessing tool (ekphrasis): tailored to Twitter, it handles tokenization, normalization, hashtag segmentation and spelling repair.
  • Word embeddings: GloVe vectors trained on a large private tweet collection.
  • Attention-based LSTMs: a 2-layer BiLSTM with attention for Subtask A, and a Siamese BiLSTM whose attention is conditioned on the topic for B–E.

Quantification is done by averaging the classifier's class probabilities (PCC). The message-level system shared first place in Subtask A, and the topic-based system came second in B, C and D. Subtask E, however, ended near the bottom, which the authors trace to the quantification step rather than the classifier. The work shows that a carefully preprocessed, purely learned pipeline can match or beat feature-heavy approaches on noisy social-media text. Both the embeddings and the tool are publicly released.

Key points

  • Unlabeled corpus: about 330M English tweets collected between 12/2012 and 07/2016. It supplied the unigram/bigram statistics for the text processor and the training data for GloVe embeddings (660K-word vocabulary, 300 dimensions).
  • Official rankings: 1/38 (tie) in Subtask A, 2/24 in B, 2/16 in C, 2/16 in D and 11/12 in E.
  • Attention ablation (mean of 10 runs on the official test set):
    • Subtask A: average recall ρ rose from 0.678 to 0.682.
    • Subtask B: ρ rose from 0.856 to 0.863.
    • The gain is consistent but small.
  • Quantification results:
    • Subtask D: PCC beat Classify & Count (KLD 0.048 vs 0.060).
    • Subtask E: PCC was much worse (EMD 0.595 vs 0.359).
    • Suspected causes are the larger number of classes in E and a sharp shift in the positive/negative ratio between the training and test data.
  • Implementation: Keras/Theano on a single GTX750Ti, with hyper-parameters chosen by Bayesian optimization. Regularization combined Gaussian noise on the embeddings, several dropout layers and L2. Class weights in the loss offset label imbalance. Source code is shared on GitHub.

Interesting ideas

  • I1. Mark information instead of deleting it. Rather than discarding signals such as all-caps writing, elongated or repeated characters, emphasis asterisks, emoticons or hashtags, the preprocessor wraps or replaces them with special tags (e.g. <allcaps>, <hashtag>…</hashtag>). This shrinks the vocabulary while keeping cues that matter for sentiment, and it would carry over well to noisy Ukrainian social-media text.
  • I2. Corpus-driven Viterbi for segmentation and spelling. Unigram and bigram counts from the in-domain unlabeled corpus drive a Viterbi search for both hashtag splitting and spell correction. The same statistics can therefore serve two cleanup jobs cheaply, with no hand-made dictionary.
  • I3. Topic-conditioned attention. A mean-pooled encoding of the topic is concatenated onto every word state of the message before attention weights are computed. This is a simple, general way to make a sequence model focus differently depending on an external target, entity or aspect.
  • I4. Quantification from probabilities. Topic-level class proportions are estimated by averaging predicted probabilities (PCC) rather than counting hard labels. The mixed D/E results are a useful warning: when label distributions drift between training and test data, both strategies should be validated before choosing one.
  • I5. Cheaper hyper-parameter search and imbalance handling. Bayesian optimization replaces grid or random search, and loss-level class weights address skewed labels. Both are low-cost choices that apply to almost any small-data classification project.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 96 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2126

@inproceedings{baziotis2017datastories,
  title = {{DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis}},
  author = {Baziotis, Christos and Pelekis, Nikos and Doulkeridis, Christos},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2126},
  url = {https://doi.org/10.18653/v1/s17-2126}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. 2001. Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies. A field guide to dynamical recurrent neural networks. IEEE Press. find
  2. Antonio Bella, Cesar Ferri, José Hernández-Orallo, and Maria Jose Ramirez-Quintana. 2010. Quantification via probability estimators. In Proceedings of ICDM. pages 737–742. find
  3. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. find
  4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2):157–166. find
  5. Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall PTR, 1st edition. find
  6. James Bergstra, Daniel Yamins, and David D. Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of ICML. pages 115–123. find
  7. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . find
  9. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research 50:723–762. find
  10. Fran¸cois Chollet. 2015. Keras. https://github.com/fchollet/keras. ↗
  11. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278–2324. find
  12. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. pages 160–167. find
  13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. pages 3111–3119. find
  14. Jan Deriu, Maurice Gonzenbach, Fatih Uzdilli, Aurélien Lucchi, Valeria De Luca, and Martin Jaggi. 2016. SwissCheese at SemEval-2016 Task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision. In Proceedings of SemEval. pages 1124–1128. find
  15. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242 . find our card
  16. Andrea Esuli and Fabrizio Sebastiani. 2015. Optimizing Text Quantifiers for Multivariate Loss Functions. ACM Trans. Knowl. Discov. Data 9(4):27:1–27:27. find
  17. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. Semeval-2016 Task 4: Sentiment analysis in Twitter. In Proceedings of SemEval. pages 1–18. find our card
  18. George Forman. 2008. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery 17(2):164–206. find
  19. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Proceedings of NIPS. pages 1019–1027. find our card
  20. Elisavet Palogiannidi, Athanasia Kolovou, Fenia Christopoulou, Filippos Kokkinos, Elias Iosif, Nikolaos Malandrakis, Haris Papageorgiou, Shrikanth Narayanan, and Alexandros Potamianos. 2016. Tweester at SemEval-2016 Task 4: Sentiment analysis in Twitter using semantic-affective model adaptation. In… find
  21. Wei Gao and Fabrizio Sebastiani. 2016. From classification to quantification in tweet sentiment analysis. Social Network Analysis and Mining 6(1). find
  22. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of ICML. pages 1310– 1318. find
  23. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of ACL. pages 42–47. find our card
  24. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and others. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825–2830. find
  25. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of ICML. pages 1319–1327. find
  26. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of EMNLP. pages 1532–1543. find
  27. Christopher Potts. 2011. Sentiment Symposium Tutorial: Tokenizing. http://sentiment.christopherpotts.net/ tokenizing.html. ↗
  28. Colin Raffel and Daniel PW Ellis. 2015. Feed-forward networks with attention can solve some long-term memory problems. arXiv preprint arXiv:1512.08756 . — [arXiv:1512.08756](https://arxiv.org/abs/1512.08756) find
  29. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskáżş, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664 . find our card
  30. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of SemEval. Vancouver, Canada. find our card
  31. Mickael Rouvier and Benoît Favre. 2016. SENSEI-LIF at SemEval-2016 Task 4: Polarity embedding fusion for robust sentiment analysis. In Proceedings of SemEval. pages 202–208. find
  32. Toby Segaran and Jeff Hammerbacher. 2009. Beautiful Data: The Stories Behind Elegant Data Solutions. "O’Reilly Media, Inc.". find
  33. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1):1929–1958. find
  34. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015. Target-dependent sentiment classification with long short term memory. CoRR, abs/1512.01100 . — [arXiv:1512.01100](https://arxiv.org/abs/1512.01100) find
  35. Theano Dev Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688. find our card
  36. Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of EMNLP. pages 606–615. find our card
  37. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT. pages 1480–1489. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baziotis-2017-datastories-semeval-2017-task-4/