Elamur Research⌕
☰
home / papers / 2015 / Document Modeling with Gated Recurrent Neural Network for Sentiment Classification

Document Modeling with Gated Recurrent Neural Network for Sentiment Classification

Summary

The paper addresses document-level sentiment classification, i.e. predicting the overall polarity or star rating of a review. The authors argue that bag-of-ngrams SVMs and Paragraph Vector treat a document as a flat collection of local word patterns and miss how sentences relate to each other, for example through contrast or cause. Their neural model works hierarchically. First, a sentence encoder (either a multi-filter CNN or an LSTM) turns each sentence's pretrained word embeddings into a fixed-size sentence vector. Then a gated recurrent network reads these sentence vectors in order and produces a single document vector, which is fed to a softmax classifier. The whole model is trained end-to-end with a cross-entropy loss. The authors evaluate on four large review datasets built from IMDB and the Yelp Dataset Challenge, using two evaluation metrics. The resulting models, Conv-GRNN and LSTM-GRNN, beat SVM, Paragraph Vector, CNN and JMARS baselines. A plain RNN at the document level performs poorly, while adding gates gives a large improvement. This makes the case for explicit two-level document composition.

Key points

  • The architecture has two composition stages: words → sentence (CNN or LSTM) and sentences → document (gated RNN). The final variants are named Conv-GRNN and LSTM-GRNN.
  • The CNN sentence encoder combines three filters of widths 1, 2 and 3 (unigrams, bigrams, trigrams). Each filter's output goes through average pooling and tanh, and the results of the filters are then averaged.
  • The document-level unit is essentially an LSTM without an output gate (the output gate is effectively always open). It uses input and forget gates over the previous hidden state and the current sentence vector.
  • Two ways of reading out the document vector are compared: taking the final hidden state (GatedNN) or averaging all hidden states (GatedNN Avg).
  • Results:
    • The models are evaluated on four IMDB/Yelp review datasets against SVM with bag-of-ngrams, Paragraph Vector, CNN and the JMARS recommender.
    • The proposed models score best across the board.
    • As a sentence encoder, LSTM outperforms the multi-filter CNN.
    • The authors release their code and datasets publicly.

Interesting ideas

  • I1. Hierarchical encoding instead of one long sequence. Building sentence vectors first and then composing them keeps each recurrent sequence short. It also matches how people read longer texts. This suits any long-document task, such as news, legal texts or customer feedback.
  • I2. Removing the output gate. Dropping the output gate from an LSTM-like unit means nothing from the sentence-level state is hidden from the document vector. This is a simple, cheaper variant worth trying wherever every input chunk should contribute to the final representation.
  • I3. Averaging hidden states as a readout. Mean-pooling the recurrent states, rather than using only the last one, gives early sentences a direct path to the output. This is an easy option to test for long inputs where the final state may underweight the beginning.
  • I4. Plain RNN versus gated RNN as a diagnostic. Comparing an ungated and a gated recurrent model on the same pipeline clearly isolates the value of gating at the document level. Such a controlled ablation is a useful template for justifying architectural choices.
  • I5. Implicit discourse modeling. Sentence relations are learned from the classification signal alone, with no discourse parser. This matters for languages or domains where discourse parsers are unavailable or do not scale to web-sized data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

156 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 139 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1167

@inproceedings{tang2015document,
  title = {{Document Modeling with Gated Recurrent Neural Network for Sentiment Classification}},
  author = {Tang, Duyu and Qin, Bing and Liu, Ting},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1167},
  url = {https://doi.org/10.18653/v1/d15-1167}
}

References

57 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, pages 238–247. find our card
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  4. Yoshua Bengio, Ian J. Goodfellow, and Aaron Courville. 2015. Deep learning. Book in preparation for MIT Press. find
  5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In EMNLP, pages 1724–1734. find
  6. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2015. Gated feedback recurrent neural networks. ICML. find
  7. Misha Denil, Alban Demiraj, Nal Kalchbrenner, Phil Blunsom, and Nando de Freitas. 2014. Modelling, visualising and summarising documents with a single convolutional neural network. preprint:1406.3830. arXiv Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J Smola, Jing Jiang, and Chong Wang. 2014.… find
  8. Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2014. Adaptive multi-compositionality for recursive neural models with applications to sentiment analysis. In AAAI, pages 1537–1543. find
  9. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. JMLR. find
  10. Gottlob Frege. 1892. On sense and reference. find
  11. Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In WebDB. find
  12. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, pages 513–520. find
  13. Andrew B Goldberg and Xiaojin Zhu. 2006. Seeing stars when there aren’t many stars: graph-based semi-supervised learning for sentiment categorization. In GraphBased Method for NLP, pages 45–52. find
  14. Alex Graves, Navdeep Jaitly, and A-R Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. IEEE. find
  15. Karl Moritz Hermann and Phil Blunsom. 2013. The role of syntax in vector space models of compositional semantics. In ACL, pages 894–904. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  17. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In NIPS, pages 2096–2104. find
  18. Rie Johnson and Tong Zhang. 2015. Effective use of word order for text categorization with convolutional neural networks. NAACL. find our card
  19. Dan Jurafsky and James H Martin. 2000. Speech & language processing. Pearson Education India. find
  20. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In ACL, pages 655–665. find our card
  21. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746– 1751. find our card
  22. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, pages 723–762. find
  23. Igor Labutov and Hod Lipson. 2013. Re-embedding words. In Annual Meeting of the Association for Computational Linguistics. find
  24. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML, pages 1188–1196. find our card
  25. Jiwei Li, Dan Jurafsky, and Eudard Hovy. 2015a. When are tree structures necessary for deep learning of representations? arXiv preprint arXiv:1503.00185. find
  26. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015b. A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057. find
  27. Jiwei Li. 2014. Feature weight tuning for recursive neural networks. Arxiv preprint, 1412.3714. find
  28. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167. find
  29. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In A-CL, pages 142–150. find our card
  30. Christopher D Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. MIT press. find
  31. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL, pages 55–60. find our card
  32. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  33. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  34. Georgios Paltoglou and Mike Thelwall. 2010. A study of information retrieval weighting schemes for sentiment analysis. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pages 1386–1395. find
  35. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115– 124. find our card
  36. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. find
  37. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In EMNLP, pages 79– 86. find our card
  38. Romain Paulus, Richard Socher, and Christopher D Manning. 2014. Global belief recursive neural networks. In NIPS, pages 2888–2896. find
  39. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  40. Lizhen Qu, Georgiana Ifrim, and Gerhard Weikum. 2010. The bag-of-opinions method for review rating prediction from sparse text patterns. In COLING, pages 913–921. find
  41. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013a. Parsing with compositional vector grammars. In ACL. find
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. find
  43. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  44. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In ACL. find
  45. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL, pages 1555–1565. find our card
  46. Duyu Tang, Bing Qin, and Ting Liu. 2015. Learning semantic representations of users and products for document level sentiment classification. In ACL, pages 1014–1023. find
  47. Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In ACL, pages 417–424. find our card
  48. Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, pages 90–94. find our card
  49. Rui Xia and Chengqing Zong. 2010. Exploring the use of word relation features for sentiment classification. In COLING, pages 1336–1344. find
  50. Liheng Xu, Kang Liu, and Jun Zhao. 2014. Joint opinion relation detection using one-class deep neural network. In COLING, pages 677–687. find
  51. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. ICML. find our card
  52. Ainur Yessenalina and Claire Cardie. 2011. Compositional matrix-space models for sentiment analysis. In EMNLP, pages 172–182. find
  53. Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615. find
  54. Yongfeng Zhang, Haochen Zhang, Min Zhang, Yiqun Liu, and Shaoping Ma. 2014. Do users rate or review?: boost phrase-level sentiment labeling with review-level sentiment classification. In SIGIR, pages 1027–1030. ACM. find
  55. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In IJCAI. find
  56. Lanjun Zhou, Binyang Li, Wei Gao, Zhongyu Wei, and Kam-Fai Wong. 2011. Unsupervised discovery of discourse relations for eliminating intra-sentence polarity ambiguities. In EMNLP, pages 162–171, . find our card
  57. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over tree structures. ICML. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/