Elamur Research⌕
☰
home / papers / 2017 / An Unsupervised Neural Attention Model for Aspect Extraction

An Unsupervised Neural Attention Model for Aspect Extraction

Paper · 2017 · Ruidan He, Wee Sun Lee, Hwee Tou Ng et al.

Summary

The paper addresses unsupervised aspect extraction for aspect-based sentiment analysis. The task is to find the aspects that review opinions target and to group related aspect terms into categories without labeled data. The authors argue that LDA-style topic models give incoherent aspects for two reasons: they treat words as generated independently, and they struggle to estimate topic mixtures for short review texts. Their alternative is Attention-based Aspect Extraction (ABAE), which works like an autoencoder over word embeddings:

  • An attention layer down-weights words that are not aspect-related and builds a weighted sentence vector.
  • That vector is reconstructed as a linear combination of a small set of learned aspect embeddings.
  • Each aspect lives in the same space as words, so it can be read off from its nearest words.

The model is evaluated on restaurant and beer review domains against k-means, LocLDA, SAS, BTM and SERBM. ABAE yields more coherent aspects and generally higher F1 for sentence-level aspect identification. This makes it a simple, scalable neural replacement for topic models.

Key points

  • Architecture.

    • The sentence context y_s is the mean of the word embeddings.
    • Word relevance is scored as a bilinear form with a learned matrix M, normalised by softmax.
    • These weights define the sentence embedding z_s, and the reconstruction is r_s = Tᵀ·p_t, using the aspect matrix T (K × d).

  • Training objective.

    • The loss is a contrastive max-margin hinge: r_s should score higher with z_s than with m randomly sampled negative sentences, each represented by its averaged embeddings.
    • An orthogonality penalty ‖T_n·T_nᵀ − I‖, weighted by λ, discourages redundant aspects.

  • Beer domain. ABAE has the best F1 on every aspect except Taste:

    • Feel 0.816
    • Smell 0.575
    • Look 0.905
    • Overall 0.725
    • Taste+Smell, merged into one aspect, 0.866

  • Attention ablation (restaurant domain). Removing attention (ABAE⁻, plain averaging) lowers F1:

    AspectABAE⁻ABAE
    Food0.7910.828
    Staff0.6930.757
    Ambience0.7030.740

  • Restaurant comparison. ABAE beats the other models on F1 for Staff and Ambience. It trails SERBM on Food despite very high precision, mostly on generic sentences that contain no specific food terms.

Interesting ideas

  • I1. Shared space for topics and words. Placing aspect vectors in the same embedding space as words makes every learned topic directly interpretable through nearest-neighbour words. The same trick could label clusters in other unsupervised settings, such as grouping trade goods or place descriptions.
  • I2. Unsupervised attention via context. Conditioning the attention on the sentence's average embedding lets the model pick out salient words without any labels. This is a cheap way to filter noise words in short texts.
  • I3. Reconstruction with negative sampling. Using a hinge loss against random negatives in place of a full generative likelihood avoids the per-document topic estimation that hurts LDA on short documents. It suits any corpus of brief texts.
  • I4. Orthogonality regulariser. The penalty on row-normalised aspect vectors is a simple, reusable way to keep learned prototypes or codebooks from collapsing into duplicates.
  • I5. Evaluation design. Evaluating with a merged label (Taste+Smell) for aspects that even humans confuse gives a fairer picture of model quality. Pairing this with an attention on/off ablation is a clean template for evaluating other unsupervised models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🏢Systems, Applications & Products in Data ProcessingLondon · United Kingdom4 works · 1,483 citations🏢Systems, Applications & Products in Data Processing (Germany)Walldorf · Germany4 works · 2,374 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works General Social Sciences · 87 works

Cite

Cite as Ruidan He, Wee Sun Lee, Hwee Tou Ng and Daniel Dahlmeier. 2017. An Unsupervised Neural Attention Model for Aspect Extraction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1036

@inproceedings{he2017unsupervised,
  title = {{An Unsupervised Neural Attention Model for Aspect Extraction}},
  author = {Ruidan He, Wee Sun Lee, Hwee Tou Ng and Daniel Dahlmeier},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1036},
  url = {https://doi.org/10.18653/v1/p17-1036}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Ying-Ju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In Proceedings of the 23rd International Conference on Computational Linguistics. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.
  3. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Morgan & Claypool publishers. find
  4. David Blei, Andrew Ng, and Michael Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research 3:993–1022.
  5. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  6. Samuel Brody and Noemie Elhadad. 2010. An unsupervised aspect-sentiment model for online reviews. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  7. Julian McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning attitudes and attributes from multi-aspect reviews. In Proceedings of the 12th IEEE International Conference on Data Mining. find
  8. Huimin Chen, Maosong Sun, Cunchao Tu, Yankai Lin, and Zhiyuan Liu. 2016. Neural sentiment classification with user and product attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
  9. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  10. Zhiyuan Chen, Arjun Mukherjee, and Bing Liu. 2014. Aspect extraction with automated prior knowledge learning. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find
  11. David Mimno, Hanna M. Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing.
  12. Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In Proceedings of Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent models of visual attention. In Advances in Neural Information Proces… find
  13. Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013. A biterm topic model for short texts. In Proceedings of the 22nd International World Wide Web Conference. find
  14. Arjun Mukherjee and Bing Liu. 2012. Aspect extraction through semi-supervised modeling. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find
  15. Yichun Yin, Furu Wei, Li Dong, Kaiming Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. find
  16. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics 37:9–27.
  17. Wayne Xin Zhao, Jing Jiang, Hongfei Yan, and Xiaoming Li. 2010. Jointly modeling aspects and opinions with a MaxEnt-LDA hybrid. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. find
  18. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  19. Li Zhuang, Feng Jing, and Xiao-Yan Zhu. 2006. Movie review mining and summarization. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management. find
  20. Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics 2.
  21. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find
  22. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  23. Ivan Titov and Ryan McDonald. 2008. Modeling online reviews with multi-grain topic models. In Proceedings of the 17th International World Wide Web Conference. find
  24. Linlin Wang, Kang Liu, Zhu Cao, Jun Zhao, and Gerard de Melo. 2015. Sentiment-aspect extraction based on restricted Boltzmann machines. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing… find
  25. Wenya Wang, Sinno J. Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  26. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Scaling up to large vocabulary image annotation. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2017-unsupervised-neural-attention-model-aspect/