Elamur Research
☰
home / papers / 2011 / Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D11-1014.pdf

Summary

Most sentiment classifiers of the time relied on bag-of-words features, which ignore word order and composition. Systems that did model composition depended on hand-built resources such as sentiment lexica, parsers and polarity-shifting rules. The authors propose semi-supervised recursive autoencoders (RAE) to address both limitations. The model starts from continuous word vectors, either randomly initialized or pre-trained with a neural language model. It repeatedly merges neighbouring vectors into phrase vectors, greedily choosing the pair that reconstructs its children best, which yields a tree and a fixed-length sentence representation without any parser. A softmax layer on each node is trained jointly with the reconstruction objective to predict a distribution over sentiment labels. The model is tested on standard benchmarks such as movie reviews and on a new experience project (EP) corpus, where readers react to confessions with five possible labels. The paper reports state-of-the-art accuracy on the standard data, outperforming the CRF system of Nakagawa et al. (2010) without external tools. On EP it predicts reaction distributions better than competitive baselines. The work matters because it learns compositional, multi-dimensional sentiment with little language-specific engineering.

Key points

  • The RAE reuses one autoencoder at every tree node and builds structure greedily from reconstruction error. A CKY-like beam search performed similarly but was much slower.
  • No sentiment lexicon, POS tagger, dependency parser or polarity rules are needed. The model learns word and n-gram polarity as a by-product.
  • The EP dataset (Potts, 2010) contains over 31,000 confessions released together with the code at www.socher.org. Readers can respond with one of five reactions: "Sorry, hugs", "You rock", "Wow, just wow", "Tehee" or "I understand".
  • On EP, the model is evaluated on two tasks: predicting the top-voted label and predicting the full label distribution. It beats the baselines on both.
  • The authors argue that the learned trees capture semantic content of the word vectors even when they are not syntactically plausible. This sets the approach apart from the supervised RNNs in Socher et al. (2010, 2011).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

261 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 248 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{socher2011semi,
  title = {{Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions}},
  author = {Socher, Richard and Pennington, Jeffrey and Huang, Eric H. and Ng, Andrew Y. and Manning, Christopher D.},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1014/}
}

References

37 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beineke, T. Hastie, C. D. Manning, and S. Vaithyanathan. 2004. Exploring sentiment summarization. In Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications. find
  2. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research., 3:993–1022. find our card
  4. Y. Choi and C. Cardie. 2008. Learning with compositional semantics as structural inference for subsentential sentiment analysis. In EMNLP. find
  5. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. find
  6. S. Das and M. Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association Annual Conference (APFA). find
  7. K. Dave, S. Lawrence, and D. M. Pennock. 2003. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of WWW, pages 519–528. find
  8. X. Ding, B. Liu, and P. S. Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the Conference on Web Search and Web Data Mining (WSDM). find
  9. J. L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2-3):195–225. find
  10. A. Esuli and F. Sebastiani. 2007. Pageranking wordnet synsets: An application to opinion mining. In Proceedings of the Association for Computational Linguistics (ACL). find
  11. C. Goller and A. Küchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of the International Conference on Neural Networks (ICNN-96). find
  12. G. Grefenstette, Y. Qu, J. G. Shanahan, and D. A. Evans. 2004. Coupling niche browsers and affect analysis for an opinion mining application. In Proceedings of Recherche d’Information Assistée par Ordinateur (RIAO). find
  13. D. Ikeda, H. Takamura, L. Ratinov, and M. Okumura. 2008. Learning to shift the polarity of words for sentiment classification. In IJCNLP. find
  14. S. Kim and E. Hovy. 2007. Crystal: Analyzing predictive opinions on the web. In EMNLP-CoNLL. find
  15. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. 2011. Learning accurate, compact, and interpretable tree annotation. In Proceedings of ACL. find
  16. Y. Mao and G. Lebanon. 2007. Isotonic Conditional Random Fields and Local Sentiment Flow. In NIPS. find
  17. P. Mirowski, M. Ranzato, and Y. LeCun. 2010. Dynamic auto-encoders for semantic indexing. In Proceedings of the NIPS 2010 Workshop on Deep Learning. find
  18. T. Nakagawa, K. Inui, and S. Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In NAACL, HLT. find
  19. B. Pang and L. Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL. find
  20. B. Pang and L. Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115–124. find
  21. B. Pang and L. Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1–135. find
  22. B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? Sentiment classification using machine learning techniques. In EMNLP. find
  23. J. W. Pennebaker, R.J. Booth, and M. E. Francis. 2007. Linguistic inquiry and word count: Liwc2007 operators manual. University of Texas. find
  24. L. Polanyi and A. Zaenen. 2006. Contextual valence shifters. find
  25. J. B. Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46:77–105, November. find
  26. C. Potts. 2010. On the negativity of negation. In David Lutz and Nan Li, editors, Proceedings of Semantics and Linguistic Theory 20. CLC Publications, Ithaca, NY. find
  27. B. Snyder and R. Barzilay. 2007. Multiple aspect ranking using the Good Grief algorithm. In HLT-NAACL. find
  28. R. Socher, C. D. Manning, and A. Y. Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop. find
  29. R. Socher, C. C. Lin, A. Y. Ng, and C. D. Manning. 2011. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. find
  30. P. J. Stone. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press. find
  31. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394. find our card
  32. P. Turney. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In ACL. find
  33. L. Velikovich, S. Blair-Goldensohn, K. Hannan, and R. McDonald. 2010. The viability of web-derived polarity lexicons. In NAACL, HLT. find
  34. T. Voegtlin and P. Dominey. 2005. Linear Recursive Distributed Representations. Neural Networks, 18(7). find
  35. J. Wiebe, T. Wilson, and C. Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39. find
  36. T. Wilson, J. Wiebe, and P. Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP. find
  37. H. Yu and V. Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In EMNLP. find