Elamur Research
☰
home / papers / 2014 / Convolutional Neural Networks for Sentence Classification

Convolutional Neural Networks for Sentence Classification

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1408.5882

Summary

Classifying single sentences by sentiment, subjectivity or question type used to rely on hand-built features or on heavy neural architectures that need parse trees. Kim tests whether a very shallow convolutional network on top of word2vec embeddings is enough. The embeddings were pre-trained on Google News. The model runs filters of widths 3, 4 and 5 (100 maps each) over the sentence's embedding matrix. Max-over-time pooling keeps each filter's strongest response, and a softmax layer with dropout and an l2-norm cap makes the final prediction. There are four variants: random embeddings, frozen word2vec, fine-tuned word2vec, and a two-channel model that pairs a frozen copy with a tuned copy. Hyperparameters were set once on the SST-2 dev set and reused unchanged on all seven benchmarks: MR, SST-1, SST-2, Subj, TREC, CR and MPQA. Pre-trained vectors gave much larger gains over random initialization than expected. The CNN variants set new best results on four of the seven tasks. The paper became a standard baseline because it showed that general-purpose embeddings plus a minimal CNN can match more complex, task-specific systems.

Key points

  • Setup: 300-dimensional word2vec (CBOW) vectors trained on about 100B Google News tokens, ReLU, dropout p = 0.5, l2 constraint s = 3, mini-batches of 50 and the Adadelta optimizer. Words missing from word2vec get random vectors drawn from U[−a, a], scaled to match the variance of the pre-trained vectors.
  • Best scores: CNN-non-static reached 81.5 on MR, 48.0 on SST-1, 93.4 on Subj and 93.6 on TREC. CNN-multichannel reached 88.1 on SST-2 and 85.0 on CR, and CNN-static reached 89.6 on MPQA. With random embeddings, CNN-rand scored only 76.1 on MR.
  • The model did not win everywhere. Paragraph-Vec (48.7 on SST-1), an SVM with hand-crafted features (95.0 on TREC) and MNB/F-Dropout (93.6 on Subj) stayed ahead.
  • The two-channel design gave mixed results compared with a single channel. Dropout added about 2–4% relative accuracy. word2vec clearly beat the Wikipedia-trained embeddings from Collobert et al. (SENNA).
  • Fine-tuning makes the embeddings carry sentiment. In the original word2vec space, "good" sits closest to "bad". After tuning on SST-2, its nearest neighbors become words like "nice" and "great". Randomly initialized tokens such as "!" also pick up useful meaning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1779 papers of our library cite it (of the 140673 whose bibliographies we read); 57 of them have a page here, 1722 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1408.5882 · arXiv 1408.5882

@misc{kim2014convolutional,
  title = {{Convolutional Neural Networks for Sentence Classification}},
  author = {Kim, Yoon},
  year = {2014},
  doi = {10.48550/arxiv.1408.5882},
  eprint = {1408.5882},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1408.5882}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, R. Ducharme, P. Vincent. 2003. Neural Probabilitistic Language Model. Journal of Machine Learning Research 3:1137–1155. find
  2. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuglu, P. Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 12:2493–2537. find our card
  3. J. Duchi, E. Hazan, Y. Singer. 2011 Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  4. L. Dong, F. Wei, S. Liu, M. Zhou, K. Xu. 2014. A Statistical Parsing Framework for Sentiment Classification. CoRR, abs/1401.6330. find
  5. A. Graves, A. Mohamed, G. Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP 2013. find
  6. G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580. find
  7. K. Hermann, P. Blunsom. 2013. The Role of Syntax in Vector Space Models of Compositional Semantics. In Proceedings of ACL 2013. find
  8. M. Hu, B. Liu. 2004. Mining and Summarizing Customer Reviews. In Proceedings of ACM SIGKDD 2004. find
  9. M. Iyyer, P. Enns, J. Boyd-Graber, P. Resnik 2014. Political Ideology Detection Using Recursive Neural Networks. In Proceedings of ACL 2014. find
  10. N. Kalchbrenner, E. Grefenstette, P. Blunsom. 2014. A Convolutional Neural Network for Modelling Sentences. In Proceedings of ACL 2014. find our card
  11. A. Krizhevsky, I. Sutskever, G. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of NIPS 2012. find
  12. Q. Le, T. Mikolov. 2014. Distributed Represenations of Sentences and Documents. In Proceedings of ICML 2014. find
  13. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 86(11):2278– 2324, November. find
  14. X. Li, D. Roth. 2002. Learning Question Classifiers. In Proceedings of ACL 2002. find
  15. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS 2013. find
  16. T. Nakagawa, K. Inui, S. Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In Proceedings of ACL 2010. find
  17. B. Pang, L. Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL 2004. find
  18. B. Pang, L. Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL 2005. find
  19. A.S. Razavian, H. Azizpour, J. Sullivan, S. Carlsson 2014. CNN Features off-the-shelf: an Astounding Baseline. CoRR, abs/1403.6382. — [arXiv:1403.6382](https://arxiv.org/abs/1403.6382) find
  20. Y. Shen, X. He, J. Gao, L. Deng, G. Mesnil. 2014. Learning Semantic Representations Using Convolutional Neural Networks for Web Search. In Proceedings of WWW 2014. find
  21. J. Silva, L. Coheur, A. Mendes, A. Wichert. 2011. From symbolic to sub-symbolic information in question classification. Artificial Intelligence Review, 35(2):137–154. find
  22. R. Socher, J. Pennington, E. Huang, A. Ng, C. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of EMNLP 2011. find our card
  23. R. Socher, B. Huval, C. Manning, A. Ng. 2012. Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of EMNLP 2012. find
  24. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng, C. Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of EMNLP 2013. find
  25. J. Wiebe, T. Wilson, C. Cardie. 2005. Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation, 39(2-3): 165– 210. find
  26. S. Wang, C. Manning. 2012. Baselines and Bigrams: Simple, Good Sentiment and Topic Classification. In Proceedings of ACL 2012. find
  27. S. Wang, C. Manning. 2013. Fast Dropout Training. In Proceedings of ICML 2013. find
  28. B. Yang, C. Cardie. 2014. Context-aware Learning for Sentence-level Sentiment Analysis with Posterior Regularization. In Proceedings of ACL 2014. find
  29. W. Yih, K. Toutanova, J. Platt, C. Meek. 2011. Learning Discriminative Projections for Text Similarity Measures. Proceedings of the Fifteenth Conference on Computational Natural Language Learning, 247–256. find
  30. W. Yih, X. He, C. Meek. 2014. Semantic Parsing for Single-Relation Question Answering. In Proceedings of ACL 2014. find
  31. M. Zeiler. 2012. Adadelta: An adaptive learning rate method. CoRR, abs/1212.5701. find our card