Elamur Research⌕
☰
home / papers / 2015 / Deep Unordered Composition Rivals Syntactic Methods for Text Classification

Deep Unordered Composition Rivals Syntactic Methods for Text Classification

Summary

The paper asks whether text classification really needs costly syntax-aware composition, such as recursive or convolutional networks over parse trees and word order, or whether nonlinear depth matters more. The authors propose the deep averaging network (DAN). It averages the word embeddings of a text, passes that average through several feed-forward nonlinear layers, and classifies the result with a softmax. They also introduce word dropout, a regularizer that removes whole tokens from the average during training.

DANs are tested on sentence- and document-level sentiment analysis and on factoid question answering. The comparisons include shallow NBOW baselines and syntactic models such as RecNN, RecNTN and CNNs. The DAN gets close to state-of-the-art accuracy and clearly beats shallow bag-of-words models. It also uses out-of-domain training data with different syntax better than RecNNs, and it trains in minutes on a single laptop core. Error analysis shows that syntactic models fail on largely the same difficult inputs, such as double negation.

The result offers a cheap, strong baseline. It also suggests that, for these tasks, pretrained embeddings and depth matter more than modelling syntax.

Key points

  • Architecture and cost. A DAN is an embedding average, followed by one or more f(W·z + b) layers, followed by a softmax.
    • Its cost grows with the number of layers, not with the number of parse-tree nodes.
    • Its training time shows no significant difference from shallow NBOW.
    • In preliminary tests, averaging worked better than summing embeddings.
  • Word dropout. Each token is kept or dropped by an independent Bernoulli draw before averaging. Applying the same idea to RecNN leaf nodes brought no significant gain for sentiment and slightly hurt QA.
  • Pretrained embeddings matter. Random initialization made DAN converge more slowly and to a worse solution, about 3% lower absolute accuracy on coarse-grained SST. Convolutional and recursive networks show the same effect.
  • Embeddings already encode sentiment. The authors probed raw 300-dimensional GloVe vectors (Common Crawl) with logistic regression on the Hu and Liu (2004) lexicon (2,006 positive and 4,783 negative words). With a balanced 3,000/1,000 train/test split, accuracy exceeded 95%.
  • Code. Available at github.com/miyyer/dan.

Interesting ideas

  • I1. Token-level dropout as cheap augmentation. Randomly removing whole words before pooling turns each training example into many variants. This could help small labelled Ukrainian datasets where overfitting is a risk.
  • I2. Probing embeddings with a lexicon. Training a simple linear classifier on lexicon words is a quick way to check whether pretrained vectors already carry a property such as sentiment. The same test could vet Ukrainian embeddings before building a downstream model.
  • I3. Separating nonlinearity from composition. Comparing a deep unordered model against syntactic ones shows how much of the gain actually comes from syntax. This makes a useful sanity baseline before investing in parser-dependent pipelines, especially for languages with less reliable parsers.
  • I4. Cascading cheap and expensive models. The authors suggest using a fast DAN by default and calling a slower syntactic model only when the DAN's confidence is low. This design could save compute in large-scale classification of trade or news data.
  • I5. Robustness to syntactic mismatch. Because a DAN does not depend on parse trees, it can mix training data whose syntax differs from the test data, as in the QA experiments. This is attractive when in-domain labelled data is scarce.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

200 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 180 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1162

@inproceedings{iyyer2015deep,
  title = {{Deep Unordered Composition Rivals Syntactic Methods for Text Classification}},
  author = {Iyyer, Mohit and Manjunatha, Varun and Boyd-Graber, Jordan and III, Hal Daumé},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1162},
  url = {https://doi.org/10.3115/v1/p15-1162}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carmen Banea, Di Chen, Rada Mihalcea, Claire Cardie, and Janyce Wiebe. 2014. Simcompass: Using deep learning word embeddings to assess cross-level similarity. In SemEval. find
  2. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of Empirical Methods in Natural Language Processing. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research. find
  4. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828. find our card
  5. Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of Empirical Methods in Natural Language Processing. find
  6. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  7. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of Empirical Methods in Natural Language Processing. find
  8. Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. Linguistic Analysis (Lambek Festschirft). find
  9. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the International Conference of Machine Learning. find
  10. George E Dahl, Ryan P Adams, and Hugo Larochelle. 2012. Training restricted boltzmann machines on word observations. In Proceedings of the International Conference of Machine Learning. find
  11. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research. find
  12. Katrin Erk and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  13. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of Empirical Methods in Natural Language Processing. find
  14. Karl Moritz Hermann, Edward Grefenstette, and Phil Blunsom. 2013. ”not not bad” is not ”bad”: A distributional account of negation. Proceedings of the ACL Workshop on Continuous Vector Space Models and their Compositionality. find
  15. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  17. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Knowledge Discovery and Data Mining. find
  18. Ozan İrsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In Proceedings of Advances in Neural Information Processing Systems. find
  19. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014a. A neural network for factoid question answering over paragraphs. In Proceedings of Empirical Methods in Natural Language Processing. find
  20. Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014b. Political ideology detection using recursive neural networks. In Proceedings of the Association for Computational Linguistics. find our card
  21. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In ACL Workshop on Continuous Vector Space Models and their Compositionality. find our card
  22. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the Association for Computational Linguistics. find our card
  23. Dimitri Kartsaklis and Mehrnoosh Sadrzadeh. 2013. Prior disambiguation of word tensors for constructing sentence vectors. In Proceedings of Empirical Methods in Natural Language Processing. find
  24. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  25. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference of Machine Learning. find our card
  26. Jiwei Li. 2014. Feature weight tuning for recursive neural networks. CoRR, abs/1412.3714. — [arXiv:1412.3714](https://arxiv.org/abs/1412.3714) find
  27. Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A recursive recurrent neural network for statistical machine translation. In Proceedings of the Association for Computational Linguistics. find
  28. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the Association for Computational Linguistics. find our card
  29. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of the Association for Computational Linguistics. find
  30. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association for Computational Linguistics. find our card
  31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing. find
  32. Asad B. Sayeed, Jordan Boyd-Graber, Bryan Rusk, and Amy Weinberg. 2012. Grammatical structures for word-level sentiment detection. In North American Association of Computational Linguistics. find
  33. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011a. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Proceedings of Advances in Neural Information Processing Systems. find
  34. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  35. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013a. Parsing With Compositional Vector Grammars. In Proceedings of the Association for Computational Linguistics. find
  36. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods in Natural Language Processing. find
  37. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1). find
  38. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of Advances in Neural Information Processing Systems. find
  39. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. find
  40. Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In Proceedings of Empirical Methods in Natural Language Processing. find
  41. Sida I. Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the Association for Computational Linguistics. find our card
  42. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In International Joint Conference on Artificial Intelligence. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/