Elamur Research⌕
☰
home / papers / 2016 / Effective LSTMs for Target-Dependent Sentiment Classification

Effective LSTMs for Target-Dependent Sentiment Classification

Summary

The paper addresses target-dependent sentiment classification: given a sentence and a target mention inside it, the goal is to predict whether the sentence is positive, negative or neutral toward that specific target. A plain sentence encoder fails here, because one sentence can praise one target and criticize another. The authors start from a standard LSTM baseline that ignores the target. They then propose two extensions. TD-LSTM runs one LSTM from the left edge toward the target and another from the right edge toward the target, then joins their final states. TC-LSTM adds a target representation that is linked to each context word during composition. The models train end-to-end with cross-entropy and use pre-trained word embeddings. They are evaluated on the Twitter benchmark of Dong et al. (2014) against feature-based SVM, adaptive recursive neural networks and a lexicon-enhanced neural model. The plain LSTM performs poorly, while the target-aware variants reach state-of-the-art accuracy. They do this without a syntactic parser or sentiment lexicon, which makes the approach purely data-driven and easy to port.

Key points

  • Evaluation uses the Twitter target-dependent dataset from Dong et al. (2014) with three classes: positive, negative and neutral. Baselines are SVM (Jiang et al., 2011), adaptive recursive NN (Dong et al., 2014) and Vo and Zhang (2015).
  • In TD-LSTM, LSTM_L reads the preceding context plus the target from left to right. LSTM_R reads the following context plus the target from right to left. Their last hidden vectors are concatenated and passed to softmax.
  • The pre-trained word vectors come from two sources: GloVe-style embeddings (Pennington et al., 2014) and sentiment-specific embeddings (Tang et al., 2014).
  • Error analysis: 85.4% of the cases that both target-aware models get wrong involve the neutral class. Positive-versus-negative confusions are rare.
  • A soft-attention variant of TD-LSTM was also tried. It scored around 65%, slightly below the plain LSTM. The authors suspect it had too many parameters for the small corpus.
  • Code was released publicly (ir.hit.edu.cn/~dytang).

Interesting ideas

  • I1. Make the target the last token each encoder sees. Two LSTMs run inward from both sentence edges and end on the target. This forces the final states to summarize the context relative to the target. It is a cheap way to make any sequence encoder entity- or aspect-aware.
  • I2. Inject the target vector into every step. TC-LSTM pairs each context word with a target representation as it is composed. This gives the model explicit target–word interaction without a parser. The trick also fits other conditioned tasks, such as stance detection or relation classification.
  • I3. Report negative results for fancier components. The attention extension hurt performance on small data, and the authors say so. This is a useful reminder to check model capacity against dataset size before adding mechanisms.
  • I4. Analyse errors by class. Breaking failures down by class showed that the neutral category is the main bottleneck. This suggests effort should go into the neutral/subjective boundary, through better labels or more data, rather than into architecture changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

139 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tang2016effective,
  title = {{Effective LSTMs for Target-Dependent Sentiment Classification}},
  author = {Tang, Duyu and Qin, Bing and Feng, Xiaocheng and Liu, Ting},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1311/}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR. find our card
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  4. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from rst discourse parsing. In EMNLP, pages 2212–2218. find our card
  5. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In ACL, pages 49–54. find our card
  6. Gottlob Frege. 1892. On sense and reference. find
  7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  8. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In NIPS, pages 2096–2104. find
  9. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. ACL, 1:151–160. find
  10. Dan Jurafsky and James H Martin. 2000. Speech & language processing. Pearson Education India. find
  11. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In ACL, pages 655–665. find our card
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751. find our card
  13. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML, pages 1188–1196. find our card
  14. Jiwei Li, Dan Jurafsky, and Eudard Hovy. 2015a. When are tree structures necessary for deep learning of representations? EMNLP. find
  15. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015b. A hierarchical neural autoencoder for paragraphs and documents. ACL. find
  16. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167. find
  17. Christopher D Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. MIT press. find
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  19. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. find
  20. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In EMNLP, pages 79–86. find our card
  21. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  22. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Y Ng. 2013a. Reasoning with neural tensor networks for knowledge base completion. NIPS. find
  23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. find
  24. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. IJCAI. find
  25. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In ACL. find
  26. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL, pages 1555–1565. find our card
  27. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. EMNLP, pages 1422–1432. find our card
  28. Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. IJCAI. find
  29. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  30. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over tree structures. ICML. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/