Recurrent Attention Network on Memory for Aspect Sentiment Analysis

Summary

The paper addresses aspect-level sentiment classification: deciding whether a review or comment is negative, neutral or positive towards one specific opinion target, when a sentence may mention several targets with different polarities. The authors argue that TD-LSTM-style models lose sentiment cues that are far from the target, and that a single attention blurs several scattered cues into one average. Their model, RAM, has four main stages. A deep bidirectional LSTM turns word embeddings into a memory. That memory is reweighted by each word's distance to the target, so every target gets its own memory. Several attention passes then read from it, and a GRU combines their outputs non-linearly before a softmax classifier. They test RAM on SemEval 2014 laptop and restaurant reviews, a Twitter set, and a new Chinese news-comment dataset where people are the targets. RAM consistently beats the state-of-the-art baselines on all four. This suggests that recurrent multi-hop attention over a target-specific memory handles contrastive, comparative and long-distance constructions better.

Key points

RAM has five modules: input embeddings (e.g. GloVe or CBOW), a DBLSTM memory, a position-weighted memory, a recurrent attention module (several attention layers fused by a GRU), and a softmax output.
Evaluation uses four datasets:
- SemEval 2014 Laptop and Restaurant reviews,
- tweets collected by Dong et al. (2014),
- a Chinese news-comment corpus with person mentions as targets, built to check how sensitive the model is to language.
In the embedding study (Table 4), frozen pre-trained embeddings (RAM-3AL-NT) scored best everywhere: Laptop 0.7449, Restaurant 0.8023, Tweet 0.6936, Comment 0.7389.
Randomly initialised, tuned embeddings (RAM-3AL-T-R) were clearly worse, for example 0.5806 on Laptop. Fine-tuning pre-trained vectors (RAM-3AL-T) landed in between.
The authors explain the embedding results by three factors: small training sets, many test words unseen in training, and overfitting once embedding parameters are added.

Interesting ideas

I1. Per-target memory through distance weighting: Encode the sentence once, then scale each memory slot by its distance to the target. This gives each target its own view of the sentence cheaply, which suits any task with several entities per sentence, such as relation or stance classification.
I2. Fusing attention hops with a GRU: Feeding the output of each attention pass into a GRU lets the model keep, forget or transform evidence across hops. Simply summing the hops cannot do this, and the design could carry over to multi-hop reading or QA models.
I3. Reading attention over contextual states: Attending over BiLSTM states rather than raw word vectors lets one memory slot capture a phrase like "not wonderful enough". This is a useful choice wherever negation or intensifiers change meaning.
I4. Freezing embeddings on small datasets: Keeping pre-trained vectors fixed keeps unseen test words in the same space as training words. This is a practical default for low-resource settings, such as small Ukrainian corpora.
I5. A simplified model for interpretation: For visualisation, the authors drop the BLSTM so that attention weights map directly onto words. Interpreting a cut-down variant like this is a neat way to sanity-check what attention focuses on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

118 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1047

@inproceedings{chen2017recurrent,
  title = {{Recurrent Attention Network on Memory for Aspect Sentiment Analysis}},
  author = {Chen, Peng and Sun, Zhongqian and Bing, Lidong and Yang, Wei},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1047},
  url = {https://doi.org/10.18653/v1/d17-1047}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Orestes Appel, Francisco Chiclana, Jenny Carter, and Hamido Fujita. 2016. A hybrid approach to the sentiment analysis problem at the sentence level. Knowl.-Based Syst., 108:110–124. find
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
Conclusions and Future Work Peng Chen, Bing Xu, Muyun Yang, and Sheng Li. 2016. Clause sentiment identification based on convolutional neural network with context embedding. find
In this paper, we proposed a framework to identify the sentiment of opinion targets. The model Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. find
In Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 2016 12th International Conference on, pages 1532–1538. find
KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259. find our card
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 27–35. find our card
Lingjia Deng and Janyce Wiebe. 2015. Joint prediction for entity/event-level sentiment analysis using probabilistic soft logic models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. CoRR, abs/1509.00685. find
Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. A holistic lexicon-based approach to opinion mining. In Proceedings of the 2008 international conference on web search and data mining, pages 231–240. find
Richard Socher, Christopher D Manning, and Andrew Y Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, pages 1–9. find
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find our card
Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, pages 151– 161. find our card
Murthy Ganapathibhotla and Bing Liu. 2008. Mining opinions in comparative sentences. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 241–248. find
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. CoRR, abs/1410.5401. find
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find
Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 151–160. find
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015. Target-dependent sentiment classification with long short term memory. CoRR, abs/1512.01100. — [arXiv:1512.01100](https://arxiv.org/abs/1512.01100) find
Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and understanding recurrent networks. CoRR, abs/1506.02078. find our card
Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. 2014. Nrc-canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 437– 442. find our card
Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find our card
Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285. find our card
Yequan Wang, Minlie Huang, xiaoyan zhu, and Li Zhao. 2016. Attention-based lstm for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find our card
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
Wayne Xin Zhao, Jing Jiang, Hongfei Yan, and Xiaoming Li. 2010. Jointly modeling aspects and opinions with a maxent-lda hybrid. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 56–65. find
Thien Hai Nguyen and Kiyoaki Shirai. 2015. Phrasernn: Phrase recursive neural network for aspect-based sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2017-recurrent-attention-network-memory-aspect/

📄 Recurrent Attention Network on Memory for Aspect Sentiment Analysis