Elamur Research⌕
☰
home / papers / 2016 / Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis

Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis

Summary

The paper addresses co-extraction of explicit aspect terms and opinion terms from review sentences, a central subtask of aspect-based sentiment analysis. Earlier approaches either relied on hand-written syntactic rules limited to particular POS tags, or on CRFs built over manually engineered features that combine only linearly. The authors introduce Recursive Neural Conditional Random Fields (RNCRF). A dependency-tree recursive neural network (DT-RNN) first computes a contextual vector for every word, using a separate matrix for each dependency relation. A linear-chain CRF then labels the words with BIO tags covering both aspects and opinions. Both parts are trained together through maximum likelihood and backpropagation, so the CRF's label signal shapes the learned representations. Hand-crafted features can optionally be added. On SemEval 2014 task 4 (subtask 1), the model is evaluated on the restaurant and laptop domains. It outperforms several baselines, the challenge's winning systems and a re-implemented LSTM. This suggests that encoding syntactic links between aspects and opinions inside the representation reduces the need for rules and feature engineering.

Key points

  • The task is framed as sequence tagging with five labels: BA, IA, BO, IO and O. Polarity prediction is left out of scope.
  • RNCRF reaches F1 of 84.05 (aspect) and 80.93 (opinion) on restaurants, and 76.83 and 76.76 on laptops. With all extra features (RNCRF+F), these rise to 84.93/84.11 and 78.42/79.44.
  • In the ablations, a softmax on top of the DT-RNN instead of a CRF loses more than 10 points on aspects. A CRF over word2vec without the DT-RNN is also weaker, especially on laptops (63.62/56.96).
  • A re-implemented LSTM with the same embeddings and labels trails RNCRF on aspect extraction by 2.90% (restaurant) and 4.10% (laptop).
  • RNCRF-O is a variant trained without opinion labels. It does worse than full RNCRF, but with simple linguistic features it still beats the best challenge system by 0.24% (restaurant) and 2.71% (laptop).
  • Among added features, name lists helped aspects the most and the sentiment lexicon helped opinions the most.
  • Embedding sizes from 25 to 400 (in steps of 25) gave fairly stable aspect results across both domains.

Interesting ideas

  • I1. Relation-specific composition matrices on a dependency tree. Each dependency relation gets its own weight matrix, so a word's vector depends on how its neighbours attach to it. This could suit any word-level tagging task where syntactic role matters, such as NER or term extraction in morphologically rich languages.
  • I2. Training the neural encoder and the CRF jointly rather than in a pipeline. Gradients from the structured output layer flow back into the representation learner. The features are therefore tuned for the labeling goal instead of being fixed beforehand, which is a reusable pattern for other neural-plus-structured models.
  • I3. Labeling two related target types in one tag set. Aspects and opinions are tagged together, so their mutual dependencies can reinforce each other. The RNCRF-O ablation, which drops one label type, offers a clean way to test whether such joint supervision actually helps.
  • I4. A slot for hand-crafted features alongside learned ones. POS tags, name lists and lexicons are appended to the neural features. This lets cheap domain resources, such as a sentiment lexicon for a low-resource language, give targeted gains without redesigning the model.
  • I5. Per-component and per-feature ablations plus a dimension sweep. Swapping the CRF for softmax, removing the tree encoder, adding one feature type at a time and varying embedding size isolate where the gains come from. This is a solid evaluation template for any hybrid model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1059

@inproceedings{wang2016recursive,
  title = {{Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis}},
  author = {Wang, Wenya and Pan, Sinno Jialin and Dahlmeier, Daniel and Xiao, Xiaokui},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1059},
  url = {https://doi.org/10.18653/v1/d16-1059}
}

References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In ACL, pages 49–54. find our card
  2. Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. 2009. The difficulty of training deep architectures and the effect of unsupervised pre-training. In AISTATS, pages 153– 160. find
  3. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, pages 97–110. find
  4. Christoph Goller and Andreas Küchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In ICNN, pages 347– 352. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  5. Minqing Hu and Bing Liu. 2004a. Mining and summarizing customer reviews. In KDD, pages 168–177. find
  6. Minqing Hu and Bing Liu. 2004b. Mining opinion features in customer reviews. In AAAI, pages 755–760. Ozan İrsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In EMNLP, pages 720–728. Mohit Iyyer, Jordan L. Boyd-Graber, Leonardo Max Batista Claudino, Richard Socher, an… find
  7. Niklas Jakob and Iryna Gurevych. 2010. Extracting opinion targets in a single- and cross-domain setting with conditional random fields. In EMNLP, pages 1035–1045. find our card
  8. Wei Jin and Hung Hay Ho. 2009. A novel lexicalized hmm-based learning framework for web opinion mining. In ICML, pages 465–472. find
  9. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In ACL, pages 655–665. find our card
  10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751. find our card
  11. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In ACL, pages 423–430. find our card
  12. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pages 282–289. find
  13. Himabindu Lakkaraju, Richard Socher, and Christopher D. Manning. 2014. Aspect specific sentiment analysis using hierarchical deep learning. In NIPS Workshop on Deep Learning and Representation Learning. find
  14. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML, pages 1188–1196. find our card
  15. Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Ying-Ju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In COLING, pages 653–661. find
  16. Kang Liu, Liheng Xu, and Jun Zhao. 2012. Opinion target extraction using word-based translation model. In EMNLP-CoNLL, pages 1346–1356. find
  17. Kang Liu, Liheng Xu, Yang Liu, and Jun Zhao. 2013a. Opinion target extraction using partially-supervised word alignment model. In IJCAI, pages 2134–2140. find
  18. Qian Liu, Zhiqiang Gao, Bing Liu, and Yuanlin Zhang. 2013b. A logic programming approach to aspect extraction in opinion mining. In WI, pages 276–283. find
  19. Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In EMNLP, pages 1433–1443. find our card
  20. Bing Liu. 2011. Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data. find
  21. Second Edition. Data-Centric Systems and Applications. Springer. Yue Lu, Malu Castellanos, Umeshwar Dayal, and ChengXiang Zhai. 2011. Automatic construction of a context-aware sentiment lexicon: An optimization approach. In WWW, pages 347–356. find
  22. Tengfei Ma and Xiaojun Wan. 2010. Opinion target extraction in Chinese news comments. In COLING, pages 782–790. find
  23. Julian McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning attitudes and attributes from multi-aspect reviews. In ICDM, pages 1020–1025. find
  24. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In SIGIR, pages 43–52. find
  25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
  26. Nir Ofek, Soujanya Poria, Lior Rokach, Erik Cambria, Amir Hussain, and Asaf Shabtai. 2016. Unsupervised commonsense knowledge enrichment for domain-specific sentiment analysis. Cognitive Computation, 8(3):467–477. find
  27. Naoaki Okazaki. 2007. CRFsuite: a fast implementation of conditional random fields (CRFs). http://www.chokkan.org/software/crfsuite/. ↗
  28. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2). find
  29. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In SemEval, pages 27–35. find our card
  30. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In EMNLP, pages 339–346. find our card
  31. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1):9–27. find our card
  32. Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2010. Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks. In NIPS, pages 1–9. find
  33. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011a. Parsing natural scenes and natural language with recursive neural networks. In ICML, pages 129–136. find
  34. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In EMNLP, pages 151– 161. find our card
  35. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In EMNLP, pages 1201–1211. find our card
  36. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. find
  37. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. TACL, 2:207–218. find our card
  38. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015. Target-dependent sentiment classification with long short term memory. CoRR, abs/1512.01100. — [arXiv:1512.01100](https://arxiv.org/abs/1512.01100) find
  39. Ivan Titov and Ryan T. McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In ACL, pages 308–316. find our card
  40. Zhiqiang Toh and Wenting Wang. 2014. DLIREC: Aspect term extraction and term polarity classification system. In SemEval, pages 235–240. find
  41. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL, pages 173–180. find our card
  42. Hao Wang and Martin Ester. 2014. A sentiment-aligned topic model for product aspect rating prediction. In EMNLP, pages 1192–1202. find
  43. Hongning Wang, Yue Lu, and ChengXiang Zhai. 2011. Latent aspect rating analysis without aspect keyword supervision. In KDD, pages 618–626. find
  44. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In ACL-IJCNLP, pages 323– 333. find
  45. Yuanbin Wu, Qi Zhang, Xuanjing Huang, and Lide Wu. 2009. Phrase dependency parsing for opinion mining. In EMNLP, pages 1533–1541. find
  46. Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In IJCAI, pages 2979–2985. find
  47. Lei Zhang, Bing Liu, Suk Hwan Lim, and Eamonn O’Brien-Strain. 2010. Extracting and ranking product features in opinion documents. In COLING, pages 1462–1470. find
  48. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In CIKM, pages 43–50. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2016-recursive-neural-conditional-random-fields/