📄 Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts

Summary

The paper addresses sentiment classification of very short texts, such as single sentences and tweets, where little context is available and bag-of-words features fall short. The authors introduce CharSCNN (Character to Sentence Convolutional Neural Network). It is a feed-forward model with two convolutional layers. The first builds a word representation from its characters, and the second combines word windows into a sentence vector. Both layers use max pooling, so inputs of any length map to fixed-size vectors. Character-derived features are concatenated with word embeddings, which can be initialized from unsupervised pre-training in the word2vec style (Mikolov et al., 2013). The model is evaluated on two corpora: the Stanford Sentiment Treebank (SSTb, movie-review sentences) and the Stanford Twitter Sentiment corpus (STS). The authors report state-of-the-art accuracy on both: 85.7% binary and 48.3% fine-grained on SSTb, and 86.4% on STS. The work matters because it matches or beats the parser-dependent RNTN without needing any syntactic input.

Key points

• On SSTb fine-grained (five-class) prediction, CharSCNN improves on RNTN (Socher et al., 2013b) by 2.6 absolute points. Binary accuracy is about the same for CharSCNN, SCNN (the word-only variant) and RNTN.
• On STS, CharSCNN reaches 86.4 with pre-trained embeddings, compared with:
  • SCNN: 85.2
  • LProp (Speriosu et al., 2011): 84.7
  • MaxEnt, NB and SVM from Go et al. (2009): 83.0, 82.7 and 82.2
• Unsupervised pre-training of word embeddings adds about 1.5 accuracy points on SSTb and about 4.5 on STS, compared with random initialization.
• Character-level features help most on tweets, adding 1.2 points over SCNN on STS when pre-training is used.
• Implementation and cost:
  • The model is built in Theano, with backpropagation done through automatic differentiation.
  • Training uses SGD on negative log-likelihood with a softmax output.
  • One epoch on SSTb (all phrases, five classes) takes roughly 10 minutes on a 4-thread Xeon machine.

Interesting ideas

• I1. Character convolution with max pooling as a word encoder. Running a convolution over character windows and keeping the maximum per filter gives a fixed-length vector for any word. This captures affixes like "-ly" and pieces of hashtags such as "#SoSad". It could help morphologically rich languages like Ukrainian, or noisy user text full of rare forms.
• I2. Reusing one pooling pattern at two levels. The same convolution-plus-max idea is applied first to characters within words and then to words within sentences. This gives a simple, parser-free way to handle variable-length input at every level. It is attractive where good syntactic parsers are unavailable.
• I3. Interpreting the model by counting pooling winners. The authors count how many of the 300 max-pooled sentence features come from each word. This reveals what the model focuses on: in their example, "definitely not dull" accounts for 77% of the features. It is a cheap diagnostic for checking negation handling or other phenomena in any max-pooled CNN.
• I4. Running ablations along two axes. Comparing with and without character features, and with pre-trained versus random embeddings, cleanly separates the contribution of each component per domain. The ablations show both components matter much more for tweets, which is a useful evaluation template for new datasets.
• I5. Training on labeled phrases, not just full sentences. The authors suggest that sentiment-labeled phrases teach the model how parts combine into a sentence's sentiment, though they say this still needs confirmation. The idea may be worth testing wherever treebank-style phrase annotations exist.