Elamur Research⌕
☰
home / papers / 2015 / Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings

Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings

Summary

The paper addresses opinion target (aspect term) extraction, one of the fine-grained opinion mining tasks. The authors treat it as token-level BIO tagging and ask whether recurrent neural networks can replace carefully engineered CRF systems. They compare Elman-type, Jordan-type and LSTM RNNs, plus bi-directional variants. Each model reads a window of concatenated word vectors and classifies every token with a softmax layer. The embedding table is initialized from SENNA, Google or Amazon vectors, or randomly, and is then fine-tuned during training. A further variant feeds simple POS and chunk features into the network. Experiments use the SemEval-2014 Laptop and Restaurant datasets.

With word embeddings as their only input, the RNNs beat feature-rich CRFs and come second among SemEval-2014 participants on both domains. Adding the cheap linguistic features lets LSTM take first place on Laptop and second on Restaurant. The work matters because it shows that competitive opinion extraction is possible without expensive, task-specific feature pipelines. The code is released publicly.

Key points

  • Task and data: aspect term extraction framed as BIO sequence labeling, evaluated on the SemEval-2014 Laptop and Restaurant datasets. Results are reported on the standard test split and with 10-fold cross-validation.
  • Models compared: Elman-RNN, Jordan-RNN and LSTM, each with or without bi-directionality and extra features. All take a context window (e.g. m = 3) of embeddings as input.
  • Embeddings: pre-trained vectors come from SENNA, Google and Amazon. Fine-tuning SENNA in an Elman RNN added 13.01% absolute on Laptop and 4.11% on Restaurant. Untuned Amazon vectors did worse than random initialization because training got stuck in a local minimum.
  • Linguistic features: POS and chunk tags gave up to 1.25% (Laptop) and 1.44% (Restaurant) on the 10-fold setup. These gains were significant (p < 0.004 and p < 6×10⁻⁵).
  • Comparison with SemEval-2014: LSTM+Feat. beat the top system on Laptop and ranked second on Restaurant. The leading systems were IHS RD and DLIREC, both CRF-based with heavy features such as dependencies, named entities and word clusters.

Interesting ideas

  • I1. Always fine-tune pre-trained embeddings. Updating the vectors on the task gave large gains and pulled even weak starting points (random, Amazon) close to the best ones. When transferring embeddings to a new domain or a lower-resource language, fine-tuning may matter more than which embedding source you pick.
  • I2. Check data properties before adding model complexity. Aspect terms are short (about 1.4 tokens on average, versus 3.3 for MPQA expressions). This explains why bi-directional and LSTM models gave no clear benefit over a plain Elman RNN with a context window. Measuring span lengths first can save effort on unnecessary architectures.
  • I3. Share weights across both directions. Using the same parameters for the forward and backward passes keeps a bi-directional model the same size as a uni-directional one. This is a cheap guard against overfitting on small labeled datasets.
  • I4. Layer cheap linguistic signals on top of embeddings. POS and chunk tags are easy to produce, yet they gave consistent, significant improvements. Inexpensive tagger outputs can be a useful complement to learned representations, especially where full parsers are unavailable or costly.
  • I5. Use a context window as the input unit. Concatenating the vectors of neighboring words captures short-range dependencies directly. A simple recurrent layer can then handle the remaining context, which makes this a lightweight design for span-tagging tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

100 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1168

@inproceedings{liu2015fine,
  title = {{Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings}},
  author = {Liu, Pengfei and Joty, Shafiq and Meng, Helen},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1168},
  url = {https://doi.org/10.18653/v1/d15-1168}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. find
  2. Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying expressions of opinion in context. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, pages 2683–2688. Morgan Kaufmann Publishers Inc. find
  3. Maryna Chernyshevich. 2014. IHS R&D Belarus: Cross-domain extraction of product features using conditional random fields. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), page 309. find
  4. Yejin Choi and Claire Cardie. 2010. Hierarchical sequential learning for extracting opinions and their attributes. In Proceedings of the ACL 2010 Conference Short Papers, pages 269–274. ACL. find
  5. Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. Identifying sources of opinions with conditional random fields and extraction patterns. In Proceedings of HLT/EMNLP, pages 355– 362. ACL. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. ACM. find
  7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  8. Nguyen Viet Cuong, Nan Ye, Wee Sun Lee, and Hai Leong Chieu. 2014. Conditional random field with high-order dependencies for sequence labeling and segmentation. The Journal of Machine Learning Research, 15(1):981–1009. find
  9. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  10. Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In Proceedings of ICML, pages 1764–1772. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  12. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of SIGKDD, pages 168–177. ACM. find
  13. Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of EMNLP, pages 720–728. find
  14. Wei Jin, Hung Hay Ho, and Rohini K Srihari. 2009. A novel lexicalized HMM-based learning framework for web opinion mining. In Proceedings of ICML, pages 465–472. Citeseer. find
  15. Michael I Jordan. 1997. Serial order: A parallel distributed processing approach. Advances in psychology, 121:471–495. find
  16. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of ICML, pages 282–289. find
  17. Phong Le and Willem Zuidema. 2015. Compositional distributional semantics with long short term memory. In Proceedings of the joint Conference on Lexical and Computational Semantics (*SEM). find
  18. Rémi Lebret and Ronan Lebret. 2013. Word emdeddings through Hellinger PCA. arXiv preprint arXiv:1312.5542. — [arXiv:1312.5542](https://arxiv.org/abs/1312.5542) find
  19. Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceedings of CIKM, pages 375–384. ACM. find
  20. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172. ACM. find
  21. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In Proceedings of IN-TERSPEECH, pages 3771–3775. find
  22. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH, pages 1045–1048. find
  23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  25. Tomas Mikolov, 2012. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology. find
  26. Samaneh Moghaddam and Martin Ester. 2012. On the design of LDA models for aspect-based opinion mining. In Proceedings of CIKM, pages 803–812. ACM. find
  27. Kevin Murphy. 2012. Machine Learning A Probabilistic Perspective. The MIT Press. find
  28. Maria Pontiki, Haris Papageorgiou, Dimitrios Galanis, Ion Androutsopoulos, John Pavlopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35. find our card
  29. Hasim Sak, Andrew Senior, and Françoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of INTERSPEECH, pages 338– 342. find
  30. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
  31. Shabnam Shariaty and Samaneh Moghaddam. 2011. Fine-grained opinion mining using conditional random fields. In International Conference on Data Mining Workshops (ICDMW), pages 109–114. IEEE. find
  32. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631– 1642. Citeseer. find
  33. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Proceedings of INTERSPEECH, pages 194– 197. find
  34. Ivan Titov and Ryan McDonald. 2008. Modeling online reviews with multi-grain topic models. In Proceedings of WWW, pages 111–120. ACM. find
  35. Zhiqiang Toh and Wenting Wang. 2014. DLIREC: Aspect term extraction and term polarity classification system. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), page 235. find
  36. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of ACL, pages 384–394. ACL. find our card
  37. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165–210. find
  38. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of HLT/EMNLP, pages 347–354. ACL. find our card
  39. Bishan Yang and Claire Cardie. 2012. Extracting opinion expressions with semi-Markov conditional random fields. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1335–1345. ACL. find
  40. Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1640–1649. ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2015-fine-grained-opinion-mining-recurrent/