Elamur Research⌕
☰
home / papers / 2018 / Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction

Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction

Summary

The paper addresses supervised aspect extraction: finding the product attributes that reviewers comment on, such as "speed" in a laptop review. The authors aim to match strong results without handcrafted features, lexicons, or heavy architectures. Their model, DE-CNN, concatenates two frozen pre-trained embeddings for each word. One is a general-purpose GloVe embedding. The other is a domain embedding trained with fastText on a small corpus that exactly matches the target domain (Amazon laptop reviews or Yelp restaurant reviews). The combined vectors pass through four 1D convolutional layers. These layers use zero padding and stride 1 and omit max-pooling, so each output stays aligned with its input token. A shared fully-connected layer with softmax then assigns B/I/O tags. On the SemEval-2014 laptop and SemEval-2016 restaurant benchmarks, DE-CNN beats systems that use extra supervision such as opinion labels. The work shows that a simple, parallelizable CNN with well-chosen embeddings can rival complex LSTM- and attention-based systems. This matters for fast inference in real applications.

Key points

  • Data: SemEval-2014 Task 4 laptop set (3045/800 train/test sentences) and SemEval-2016 Task 5 restaurant set (2000/676 train/test sentences).
  • Embeddings: glove.840B.300d serves as the general embedding. The domain embeddings are 100-dimensional fastText vectors, with subword n-grams covering OOV words. Both embedding layers are kept non-trainable.
  • Results: DE-CNN reaches F1 81.59 on laptops and 74.37 on restaurants (significant at 0.05). It outperforms RNCRF, CMLA, MIN and the NER-oriented BiLSTM-CNN-CRF.
  • Ablations show that each design choice helps:
    • Using only one embedding type (GloVe-CNN, Domain-CNN) performs worse.
    • Adding max-pooling hurts.
    • Out-of-domain corpora (all electronics or all Yelp reviews) are poorer substitutes for in-domain ones.
    • Smaller GoogleNews embeddings perform worse than GloVe.
    • A CRF output layer brings no gain.
  • Remaining errors come mainly from inconsistent gold labels and from unseen aspects that appear in coordinations with "and".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2094

@inproceedings{xu2018double,
  title = {{Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction}},
  author = {Xu, Hu and Liu, Bing and Shu, Lei and Yu, Philip S.},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2094},
  url = {https://doi.org/10.18653/v1/p18-2094}
}

References

49 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  2. Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. 1987. Occam’s razor. Information processing letters, 24(6):377–380. find
  3. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. find
  5. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML ’01, pages 282–289. find
  6. Danushka Bollegala, Kohei Hayashi, and Ken-ichi Kawarabayashi. 2017. Think globally, embed locally—locally linear meta-embedding of words. arXiv preprint arXiv:1709.06671. — [arXiv:1709.06671](https://arxiv.org/abs/1709.06671) find
  7. Yann LeCun, Yoshua Bengio, et al. 1995. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995. find
  8. Danushka Bollegala, Takanori Maehara, and Ken-ichi Kawarabayashi. 2015. Unsupervised cross-domain word representation learning. arXiv preprint arXiv:1505.07184. find
  9. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 302–308. find
  10. Erik Cambria and Amir Hussain. 2012. Sentic Computing Techniques, Tools, and Applications 2nd Edition. find
  11. Tao Chen, Ruifeng Xu, Yulan He, and Xuan Wang. 2017. Improving sentiment analysis via sentence type classification using bilstm-crf and cnn. Expert Systems with Applications, 72:221–230. find
  12. Xin Li and Wai Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2886–2892. find
  13. Maryna Chernyshevich. 2014. Ihs r&d belarus: Cross-domain extraction of product features using crf. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 309– 313. find
  14. Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. In CIKM ’09, pages 375–384. find
  15. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers. find
  16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122. find our card
  17. Kang Liu, Liheng Xu, Yang Liu, and Jun Zhao. 2013. Opinion target extraction using partially-supervised word alignment model. In IJCAI ’13, pages 2134– 2140. find
  18. Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2017. An unsupervised neural attention model for aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 388–397. find
  19. Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433–1443. find our card
  20. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517. International World Wide Web Conferences Steering Committee. find
  21. Qiaozhu Mei, Xu Ling, Matthew Wondra, Hang Su, and ChengXiang Zhai. 2007. Topic sentiment mixture: Modeling facets and opinions in weblogs. In WWW ’07, pages 171–180. find
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  23. Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In ACL ’13, pages 1643–1654. find our card
  24. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD ’04, pages 168– 177. find
  25. Samaneh Moghaddam and Martin Ester. 2011. ILDA: interdependent lda model for learning latent aspects and their ratings from online product reviews. In SIGIR ’11, pages 665–674. find
  26. Niklas Jakob and Iryna Gurevych. 2010. Extracting opinion targets in a single- and cross-domain setting with conditional random fields. In EMNLP ’10, pages 1035–1045. find our card
  27. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2:1– 135. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  29. Zhiqiang Toh and Jian Su. 2016. Nlangp at semeval-2016 task 5: Improving aspect based sentiment analysis using neural network features. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 282–288. find
  30. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, AL-Smadi Mohammad, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th international workshop on semantic… find our card
  31. Bo Wang and Houfeng Wang. 2008. Bootstrapping both product features and opinion words from chinese customer reviews with cross-inducing. In IJC-NLP ’08, pages 289–295. find
  32. Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. arXiv preprint arXiv:1603.06679. find our card
  33. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. Associa… find our card
  34. Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In AAAI, pages 3316–3322. find
  35. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270– 280. find
  36. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In HLT-EMNLP ’05, pages 339–346. find our card
  37. Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018a. Lifelong domain word embedding via meta-learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. AAAI Press. find
  38. Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016. Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems, 108:42–49. find
  39. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1):9–27. find our card
  40. Hu Xu, Sihong Xie, Lei Shu, and Philip S. Yu. 2018b. Dual attention network for product compatibility and function satisfiability analysis. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
  41. Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 338–348, Copenhagen, Denmark. find our card
  42. Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. arXiv preprint arXiv:1605.07843. — [arXiv:1605.07843](https://arxiv.org/abs/1605.07843) find
  43. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. find our card
  44. Lei Shu, Bing Liu, Hu Xu, and Annice Kim. 2016. Lifelong-rl: Lifelong relaxation labeling for separating entities and aspects in opinion targets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  45. Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2013. Collective opinion target extraction in Chinese microblogs. In EMNLP ’13, pages 1840–1850. find
  46. Lei Shu, Hu Xu, and Bing Liu. 2017. Lifelong learning crf for supervised aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 148–154. find
  47. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In CIKM ’06, pages 43–50. find
  48. Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2670–2680. find our card
  49. Ivan Titov and Ryan McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In ACL ’08: HLT, pages 308–316. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2018-double-embeddings-cnn-based-sequence/