Elamur Research⌕
☰
home / papers / 2016 / Aspect Level Sentiment Classification with Deep Memory Network

Aspect Level Sentiment Classification with Deep Memory Network

Summary

The paper addresses aspect level sentiment classification: given a sentence and one aspect term in it, the goal is to predict the polarity toward that aspect (positive, negative, neutral). The authors argue that sequential models such as LSTM treat every context word with the same operation and do not show which words drive the decision. They propose a deep memory network inspired by Sukhbaatar et al. (2015). The context word embeddings form an external memory. The aspect vector repeatedly queries this memory through several stacked hops, and each hop combines content-based and location-based attention with a linear transform of the aspect. The final hop's output feeds a softmax classifier, and the whole model is trained end-to-end with cross-entropy. It needs no parser and no sentiment lexicon. On the SemEval 2014 laptop and restaurant datasets, the model roughly matches a strong feature-based SVM and clearly beats LSTM and attention-based LSTM. It is also much faster. This shows that lightweight, interpretable attention can compete with heavily engineered systems.

Key points

  • The experiments draw on two SemEval 2014 benchmarks, one covering laptops and one covering restaurants (Pontiki et al., 2014). Baselines are the feature-based SVM of Kiritchenko et al. (2014), LSTM, and attention-based LSTM (Tang et al., 2015a).
  • Each hop scores every memory slot with a small feed-forward network over the slot and the aspect vector, using tanh followed by softmax. The weighted sum of the memory is then added to a linear transform of the aspect.
  • Attention and linear layers share parameters across hops, so a 1-hop model and a 9-hop model have the same number of parameters. Adding hops improved accuracy on both datasets.
  • Location is defined as a word's absolute distance from the aspect, and four strategies for encoding it were tested. Using content and location together gave better context weights than content alone.
  • On CPU, the 9-hop memory network runs about 15 times faster than LSTM. In the error analysis, the main failure types were:
    • non-compositional idioms such as "to die for";
    • long multi-word aspects;
    • negation, comparison and conditional relations.

Interesting ideas

  • I1. Shared weights across hops. Tying parameters over hops lets you deepen reasoning without growing the model. This could be a cheap way to add depth to small models for low-resource languages such as Ukrainian.
  • I2. Target as a query, context as memory. Treating the target term as a query and the remaining words as memory fits any target-dependent task, such as entity-level stance or opinions about a product or a country in trade news. The attention weights also show which words drove the decision.
  • I3. Distance-aware attention. Biasing attention toward words close to the target is a simple prior that needs no syntactic parser. It is useful where parsers or UD treebank-trained tools are weak.
  • I4. Averaging multi-word aspects. Representing a multi-word aspect by averaging its word vectors is a quick baseline. The paper's error analysis also flags it as a weak point, which makes it a useful thing to check when diagnosing errors.
  • I5. Error categories as an evaluation checklist. Grouping errors into idioms, complex aspects, and negation, comparison or condition gives a practical template for evaluating sentiment systems on new corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

146 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 132 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1021

@inproceedings{tang2016aspect,
  title = {{Aspect Level Sentiment Classification with Deep Memory Network}},
  author = {Tang, Duyu and Qin, Bing and Liu, Ting},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1021},
  url = {https://doi.org/10.18653/v1/d16-1021}
}

References

41 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR). find our card
  2. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from rst discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212–2218. find our card
  3. Yejin Choi and Claire Cardie. 2010. Hierarchical sequential learning for extracting opinions and their attributes. In Proceedings of the ACL 2010 Conference Short Papers, pages 269–274. Association for Computational Linguistics. find
  4. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 49–54. find our card
  5. Gottlob Frege. 1892. On sense and reference. find
  6. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 513–520. find
  7. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  8. Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 720–728. find
  9. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 655–665. find our card
  10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751. find our card
  11. Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. 2014. Nrc-canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 437–442. find our card
  12. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. find
  13. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285. find our card
  14. Himabindu Lakkaraju, Richard Socher, and Chris Manning. 2014. Aspect specific sentiment analysis using hierarchical deep learning. In NIPS Workshop on Deep Learning and Representation Learning. find
  15. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of The 31nd International Conference on Machine Learning, pages 1188–1196. find our card
  16. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436–444. find
  17. Jiwei Li, Thang Luong, and Dan Jurafsky. 2015a. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1106–1115. find
  18. Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015b. When are tree structures necessary for deep learning of representations? In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2304–2314. find
  19. Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433–1443. find our card
  20. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167. find
  21. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. find
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  23. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  24. Thien Hai Nguyen and Kiyoaki Shirai. 2015. Phrasernn: Phrase recursive neural network for aspect-based sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2509–2514. find
  25. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. find
  26. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  27. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35. find our card
  28. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389. find
  29. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  30. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems, pages 2431–2439. find
  31. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (I-JCAI 2015), pages 1333–1339. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  33. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1556– 1566. find
  34. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015a. Target-Dependent Sentiment Classification with Long Short Term Memory. ArXiv preprint arXiv:1512.01100. — [arXiv:1512.01100](https://arxiv.org/abs/1512.01100) find
  35. Duyu Tang, Bing Qin, and Ting Liu. 2015b. Document modeling with gated recurrent neural network for sentiment classification. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432. find our card
  36. Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (I-JCAI 2015), pages 1347–1353. find
  37. Joachim Wagner, Piyush Arora, Santiago Cortes, Utsab Barman, Dasha Bogdanova, Jennifer Foster, and Lamia Tounsi. 2014. Dcu: Aspect-based polarity classification for semeval task 4. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 223–229. find
  38. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  39. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  40. Ainur Yessenalina and Claire Cardie. 2011. Compositional matrix-space models for sentiment analysis. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 172– 182. Wenpeng Yin and Hinrich Schütze. 2015. Multichannel variable-size convolution for sentence… find
  41. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over tree structures. In Proceedings of The 32nd International Conference on Machine Learning, pages 1604–1612. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/