Elamur Research
☰
home / papers / 2014 / A Convolutional Neural Network for Modelling Sentences

A Convolutional Neural Network for Modelling Sentences

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1404.2188

Summary

The paper tackles sentence modelling: turning a sentence of any length into a fixed representation that a classifier can use, without a syntactic parser or other external linguistic resources. The authors propose the Dynamic Convolutional Neural Network (DCNN). It stacks one-dimensional wide convolutions over word-embedding matrices and follows them with a new pooling operation, dynamic k-max pooling. This operation keeps the k strongest activations in their original order, and k depends on sentence length and network depth. Folding layers and multiple feature maps add to the architecture. Together, these layers build a hierarchical feature graph over the sentence that can link phrases near each other or far apart, similar to a learned parse structure. The model was tested on binary and fine-grained movie-review sentiment (Stanford Sentiment Treebank), six-way TREC question classification and Twitter sentiment learned by distant supervision. DCNN beat the strongest baselines on the sentiment tasks and matched feature-engineered systems on TREC. It needs no parse trees, so it can be applied to any language.

Key points

  • Dynamic k-max pooling generalises max pooling: it keeps the k largest values in order, and k is set from sentence length and layer position.
  • On the Stanford Sentiment Treebank, DCNN reached about 86.8% accuracy for binary sentiment and 48.5% for fine-grained sentiment, ahead of RecNN, NBoW and Max-TDNN baselines.
  • On TREC question classification, DCNN scored about 93.0% using only the labelled training data. That is not significantly different from systems built on hand-crafted features and resources.
  • For Twitter sentiment, DCNN was trained on 1.6M tweets labelled automatically by emoticons (vocabulary of 76,643 types) and tested on about 400 hand-labelled tweets. It cut the error by more than 25% compared with the best n-gram baseline from Go et al. (2009), reaching about 87.4% accuracy.
  • The authors inspected 288 first-layer filters of width 7 and found detectors for positive and negative sentiment, for negation ("not") and for intensifiers ("too").

Note: the middle of the extracted text, including the results tables, was cut off. I took the accuracy figures for the Stanford Sentiment Treebank, TREC and Twitter from my own knowledge of the paper; check them against the PDF before adding this to the library.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Treebanks

Cited by

417 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 397 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1404.2188 · arXiv 1404.2188

@misc{kalchbrenner2014convolutional,
  title = {{A Convolutional Neural Network for Modelling Sentences}},
  author = {Kalchbrenner, Nal and Grefenstette, Edward and Blunsom, Phil},
  year = {2014},
  doi = {10.48550/arxiv.1404.2188},
  eprint = {1404.2188},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1404.2188}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In EMNLP, pages 1183–1193. ACL. find our card
  2. Phil Blunsom, Krystle Kocik, and James R. Curran. 2006. Question classification with log-linear models. In SIGIR ’06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 615–616, New York, NY, USA. ACM. find
  3. Daoud Clarke. 2012. A context-theoretic framework for compositionality in distributional semantics. Computational Linguistics, 38(1):41–71. find
  4. Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical Foundations for a Compositional Distributional Model of Meaning. March. find
  5. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML. find
  6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121–2159, July. find
  7. Katrin Erk and Sebastian Padó. 2008. A structured vector space model for word meaning in context. Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP ’08, (October):897. find
  8. Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635–653. find
  9. Felix A. Gers and Jrgen Schmidhuber. 2001. Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks, 12(6):1333–1340. find
  10. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. Processing, pages 1–6. find
  11. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1394–1404. Association for Computational Linguistics. find
  12. Edward Grefenstette. 2013. Category-theoretic quantitative compositional distributional models of natural language semantics. arXiv preprint arXiv:1311.1539. — [arXiv:1311.1539](https://arxiv.org/abs/1311.1539) find
  13. Emiliano Guevara. 2010. Modelling Adjective-Noun Compositionality by Regression. ESSLLI’10 Workshop on Compositionality and Distributional Semantic Models. find
  14. Karl Moritz Hermann and Phil Blunsom. 2013. The Role of Syntax in Vector Space Models of Compositional Semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, August. Association for Computational Linguistics. Fo… find
  15. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580. find
  16. Geoffrey E. Hinton. 1989. Connectionist learning procedures. Artif. Intell., 40(1-3):185–234. find
  17. Zhiheng Huang, Marcus Thint, and Zengchang Qin. 2008. Question classification using head words and their hypernyms. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 927–936, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  18. Nal Kalchbrenner and Phil Blunsom. 2013a. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, October. Association for Computational Linguistics. find
  19. Nal Kalchbrenner and Phil Blunsom. 2013b. Recurrent Convolutional Neural Networks for Discourse Compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
  20. Dimitri Kartsaklis and Mehrnoosh Sadrzadeh. 2013. Prior disambiguation of word tensors for constructing sentence vectors. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), Seattle, USA, October. find
  21. Andreas Küchler and Christoph Goller. 1996. Inductive learning in symbolic domains using structure-driven recurrent neural networks. In Günther Görz and Steffen Hölldobler, editors, KI, volume 1137 of Lecture Notes in Computer Science, pages 183–197. Springer. find
  22. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November. find
  23. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1–7. Association for Computational Linguistics. find
  24. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In SLT, pages 234–239. find
  25. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In ICASSP, pages 5528–5531. IEEE. find
  26. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL, volume 8. find
  27. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  28. Jordan B. Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46:77–105. find
  29. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In COLING (Posters), pages 1071–1080. find
  30. Joo Silva, Lusa Coheur, AnaCristina Mendes, and Andreas Wichert. 2011. From symbolic to subsymbolic information in question classification. Artificial Intelligence Review, 35(2):137–154. find
  31. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  32. Richard Socher, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2013a. Grounded Compositional Semantics for Finding and Describing Images with Sentences. In Transactions of the Association for Computational Linguistics (TACL). find our card
  33. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1… find
  34. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  35. Peter Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. J. Artif. Intell. Res.(JAIR), 44:533–585. find
  36. Alexander Waibel, Toshiyuki Hanazawa, Geofrey Hinton, Kiyohiro Shikano, and Kevin J. Lang. 1990. Readings in speech recognition. chapter Phoneme Recognition Using Time-delay Neural Networks, pages 393–404. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. find
  37. Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar. 2010. Estimating linear models for compositional distributional semantics. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1263–1271. Association for Computational Linguis… find
  38. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI, pages 658–666. AUAI Press. find