Elamur Research⌕
☰
home / papers / 2019 / Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks

Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks

Paper · 2019 · Binxuan Huang and Kathleen Carley

Summary

The paper addresses aspect level sentiment classification, which means determining whether a sentence expresses a positive, negative or neutral attitude toward one specific aspect term. Earlier neural approaches mostly read the sentence as a flat sequence of tokens. When a sentiment word is far from its aspect in surface order, or when several modifiers could attach to several aspects, these models struggle. Huang and Carley propose TD-GAT (target-dependent graph attention network). It parses each sentence with an off-the-shelf dependency parser and then runs stacked multi-head graph attention layers over the resulting undirected graph, so that information flows from syntactic neighbours toward the aspect node. An LSTM unit carries the aspect node's state from one layer to the next. Evaluated on the SemEval 2014 laptop and restaurant datasets, the model beats several sequence-based baselines when using GloVe embeddings. It improves considerably with frozen BERT representations, and it remains far smaller and faster to train than fine-tuning BERT. The work shows that using the dependency structure directly, without binarising the tree, is an effective and inexpensive signal for this task.

Key points

  • Graph construction: the dependency graph comes from the Chen and Manning (2014) parser. A multi-word aspect is replaced by a special target symbol before parsing. The resulting meta-node's features are the average of the aspect words' embeddings.
  • Ablation: removing the cross-layer LSTM (plain GAT) lowers accuracy of the GloVe variant by about 1.2 percentage points on average.
  • Results with BERT: TD-GAT-BERT reaches roughly 80% accuracy on laptop and 83% on restaurant. It consistently outperforms BERT-AVG and BERT-CLS, and the authors found BERT fine-tuning to be unstable.
  • Model depth: a single layer is weak with GloVe, and three layers help substantially. The BERT variant is less sensitive to depth and performs best beyond 3 layers.
  • Efficiency:
    • Parameter counts are about 1.00M for TD-GAT-GloVe (3 layers) and 1.30M for TD-GAT-BERT (3 layers), versus 335.14M for BERT-CLS.
    • On restaurant, one training epoch takes about 1.15 s for TD-GAT-BERT, compared with about 226.50 s for BERT fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Sentiment & emotion · 608 works Syntax & grammar formalisms · 161 works

Cite

Cite as Binxuan Huang and Kathleen Carley. 2019. Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1549

@inproceedings{huang2019syntax,
  title = {{Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks}},
  author = {Binxuan Huang and Kathleen Carley},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1549},
  url = {https://doi.org/10.18653/v1/d19-1549}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lishuang Li, Yang Liu, and AnQiao Zhou. 2018. Hierarchical attention based position-aware network for aspect-level sentiment analysis. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 181–189. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. our card
  3. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740–750. our card
  4. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 4068–4074. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  6. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. find
  7. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers), volume 2, pages 49–54. our card
  8. Thien Hai Nguyen and Kiyoaki Shirai. 2015. Phrasernn: Phrase recursive neural network for aspect-based sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2509–2514. find
  9. Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428. our card
  10. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. find
  11. Binxuan Huang and Kathleen Carley. 2018. Parameterized convolutional neural networks for aspect level sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1091–1096. find
  12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. our card
  13. Binxuan Huang and Kathleen M Carley. 2019. Residual or gate? towards deeper graph neural networks for inductive graph representation learning. arXiv preprint arXiv:1904.08035. — [arXiv:1904.08035](https://arxiv.org/abs/1904.08035) find
  14. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. Associa… our card
  15. Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, pages 197–… find
  16. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 151–160. Association for Computational Linguist… find
  17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  18. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80. find
  19. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  20. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.09588. our card
  21. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a. Effective lstms for target-dependent sentiment classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3298– 3307. our card
  22. Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900. our card
  23. Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2018. Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. In AAAI. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. our card
  25. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. our card
  26. Joachim Wagner, Piyush Arora, Santiago Cortes, Utsab Barman, Dasha Bogdanova, Jennifer Foster, and Lamia Tounsi. 2014. Dcu: Aspect-based polarity classification for semeval task 4. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 223–229. find
  27. Yequan Wang, Minlie Huang, Li Zhao, et al. 2016. Attention-based lstm for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606–615. our card
  28. Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232. our card
  29. Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. arXiv preprint arXiv:1805.07043. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2019-syntax-aware-aspect-level-sentiment/