Elamur Research⌕
☰
home / papers / 2016 / Attention-based LSTM for Aspect-level Sentiment Classification

Attention-based LSTM for Aspect-level Sentiment Classification

Summary

The paper addresses aspect-level sentiment classification, where a single sentence can carry different polarities toward different aspects, for example praise for food alongside criticism of service. Earlier neural approaches such as TD-LSTM and TC-LSTM model a target phrase but do not use explicit aspect information. The authors give each aspect its own learned embedding and combine it with an LSTM through an attention layer, so the model weighs words differently depending on which aspect is being asked about. They propose three variants:

  • AE-LSTM adds aspect embeddings.
  • AT-LSTM uses the aspect to compute attention over hidden states.
  • ATAE-LSTM also appends the aspect vector to every word input.

Evaluation uses the SemEval 2014 benchmark (Pontiki et al., 2014), with restaurant and laptop reviews, under three-class and binary settings. The authors report state-of-the-art results over LSTM, TD-LSTM and TC-LSTM baselines. Attention visualizations show the model selecting aspect-relevant phrases. The work matters because it shows that conditioning attention on a learned aspect vector is a simple, parser-free way to separate opinions about different aspects within one sentence.

Key points

  • The data is the SemEval 2014 aspect-level dataset (Pontiki et al., 2014), with restaurant and laptop domains. The labels are positive, negative and neutral, and a binary setting drops the neutral instances.
  • The aspect-aware attention computes word weights from the LSTM hidden states concatenated with a projected aspect embedding. The final sentence vector combines the attended representation r with the last hidden state h_N, a design borrowed from Rocktäschel et al., 2015.
  • ATAE-LSTM injects the aspect embedding twice: once into each word input and once into the attention computation.
  • According to the paper, TC-LSTM scored below both plain LSTM and TD-LSTM. The authors attribute this to target vectors built by averaging word embeddings, which do not fit well with the rest of the representation.
  • Attention weights were visualized with the Heml tool. In the examples, the model picks out multi-word phrases such as "fastest delivery times" and multiple cues such as "tasteless" and "too sweet."

Interesting ideas

  • I1. Learned condition embeddings. Representing each category (here, an aspect) as a trainable vector in its own space, instead of averaging word vectors, avoids a mismatch between word and category representations. This applies to any task conditioned on a small label set, such as topic-specific stance or entity-type-aware tagging.
  • I2. Conditioning attention on a query vector. Tiling a condition vector across all time steps and concatenating it with the hidden states before scoring is a cheap way to make attention query-dependent. It transfers to targeted extraction or question-focused classification.
  • I3. Injecting the condition at input and at attention. Adding the condition to both the inputs and the attention step lets every hidden state "know" the query, not just the scoring layer. This is a design choice worth testing whenever one encoder must serve several queries.
  • I4. Residual mix of attention and last hidden state. Combining the attended summary with the final hidden state gave better results in practice. It is a small, general trick for attention-pooled classifiers.
  • I5. Parser-free modeling. The authors argue that attention handles long, convoluted sentences where parse trees are unreliable. This makes the approach attractive for lower-resource languages where parsing errors are common.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

261 papers of our library cite it (of the 140884 whose bibliographies we read); 22 of them have a page here, 239 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1058

@inproceedings{wang2016attention,
  title = {{Attention-based LSTM for Aspect-level Sentiment Classification}},
  author = {Wang, Yequan and Huang, Minlie and Zhu, Xiaoyan and Zhao, Li},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1058},
  url = {https://doi.org/10.18653/v1/d16-1058}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley,… find our card
  2. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, pages 1223–1231. find
  3. Wankun Deng, Yongbo Wang, Zexian Liu, Han Cheng, and Yu Xue. 2014. Hemi: a toolkit for illustrating heatmaps. PloS one, 9(11):e111988. find
  4. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In ACL (2), pages 49–54. find our card
  5. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  6. David Golub and Xiaodong He. 2016. Character-level question answering with attention. arXiv preprint arXiv:1604.00727. find
  7. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term… find our card
  8. Nobuhiro Kaji and Masaru Kitsuregawa. 2007. Building lexicon for sentiment analysis from massive collection of html documents. In EMNLP-CoNLL, pages 1075– 1083. find
  9. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. find
  10. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH, volume 2, page 3. find
  11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  12. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204–2212. find
  13. Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. find our card
  14. Tony Mullen and Nigel Collier. 2004. Sentiment analysis using support vector machines with diverse information sources. In EMNLP, volume 4, pages 412– 418. find
  15. Tetsuya Nasukawa and Jeonghee Yi. 2003. Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture, pages 70–77. ACM. find
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12:1532–1543. find
  17. Veronica Perez-Rosas, Carmen Banea, and Rada Mihalcea. 2012. Learning sentiment lexicons in spanish. In LREC, volume 12, page 73. find
  18. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 27–35. find our card
  19. Qiao Qian, Bo Tian, Minlie Huang, Yang Liu, Xuan Zhu, and Xiaoyan Zhu. 2015. Learning tag embeddings and tag-specific composition functions in recursive neural network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conferen… find
  20. Delip Rao and Deepak Ravichandran. 2009. Semi-supervised polarity lexicon induction. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 675–682. Association for Computational Linguistics. Tim Rocktäschel, Edward Grefenstette, Karl M… find our card
  21. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. find
  22. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computat… find our card
  23. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, volume 1631, page 1642. find
  24. Citeseer. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075. find
  25. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015a. Target-dependent sentiment classification with long short term memory. arXiv preprint arXiv:1512.01100. — [arXiv:1512.01100](https://arxiv.org/abs/1512.01100) find
  26. Duyu Tang, Bing Qin, and Ting Liu. 2015b. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432. Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. 2015. A… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/