Elamur Research⌕
☰
home / papers / 2019 / Exploiting BERT for End-to-End Aspect-based Sentiment Analysis

Exploiting BERT for End-to-End Aspect-based Sentiment Analysis

Paper · 2019 · Xin Li, Lidong Bing, Wenxuan Zhang et al.

Summary

The paper tackles End-to-End Aspect-based Sentiment Analysis (E2E-ABSA), where a system must find aspect terms in a review sentence and assign each one a sentiment at the same time. Earlier ABSA settings assumed the aspects were already given. The authors cast the task as sequence labeling with one combined tag set: B/I/E/S boundary tags merged with POS/NEG/NEU, plus O. They put pre-trained BERT ("bert-base-uncased") under several lightweight prediction heads: a linear layer, a GRU, a self-attention layer (SAN), an extra transformer layer (TFM) and a linear-chain CRF. They test these on SemEval-derived LAPTOP and REST review data, as re-prepared by Li et al. (2019a). Model selection is done consistently on a held-out development set, which most earlier work skipped. Even the plain linear head beats prior specialised E2E-ABSA systems. This suggests contextual embeddings matter more than elaborate task-specific architectures. The work offers a reproducible BERT benchmark for the task, and the code is public (BERT-E2E-ABSA on GitHub).

Key points

  • Data splits:
    • LAPTOP: 2741/304/800 sentences (train/dev/test), with 2931 aspects in total.
    • REST: 3490/387/2158 sentences, with 6593 aspects in total.
  • Training setup: BERT-base (12 layers, hidden size 768) with single-layer heads, learning rate 2e-5 and batch sizes of 25 (LAPTOP) and 16 (REST). Training ran up to 1500 steps. After step 1000, a checkpoint was chosen on the dev set every 100 steps by micro-F1. Reported scores are averages over 5 random seeds.
  • Main results:
    • BERT+Linear reached F1 60.43 on LAPTOP and 73.22 on REST, versus the strongest earlier results of 60.35 and 72.78 (Luo et al., 2019).
    • The best scores were BERT+GRU on LAPTOP (61.12) and BERT+SAN on REST (74.72).
  • Fine-tuning: Fine-tuning BERT is essential. With a frozen BERT feature extractor, F1 dropped to roughly 47–52, compared with about 73–74 when BERT was fine-tuned.
  • Overfitting: Dev-set curves indicate that the BERT-based models stay stable during longer training.

Interesting ideas

  • I1. Unified tagging: Merging span boundaries and sentiment labels into one tag set turns a two-stage pipeline into a single tagging problem. The same trick could suit any task where spans need an attribute, such as entities with types or product mentions with trade categories.
  • I2. Simple-head baselines: Testing a strong pre-trained encoder with progressively richer heads, starting from a bare linear layer, shows how much of the gain comes from the encoder itself. This is a cheap sanity check before building complex architectures, for example for Ukrainian models.
  • I3. Layer normalization in GRU gates: The authors add layer normalization inside the GRU gate computations when stacking it on transformer outputs, to avoid unstable training. This is a handy fix whenever recurrent layers are placed on top of BERT-like encoders.
  • I4. Stricter evaluation protocol: Checkpoints are selected on a separate dev set and results are averaged over several seeds, rather than tuned on test. This makes comparisons on small datasets more honest and is worth adopting as a standard practice.
  • I5. Frozen vs. fine-tuned ablation: Comparing a frozen encoder against a fine-tuned one quickly shows whether task adaptation is needed. This matters when choosing between cheap feature extraction and full fine-tuning on limited compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations🏢Alibaba DAMO AcademyHangzhou · China28 works · 6,115 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 608 works

Cite

Cite as Xin Li, Lidong Bing, Wenxuan Zhang and Wai Lam. 2019. Exploiting BERT for End-to-End Aspect-based Sentiment Analysis. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5505

@inproceedings{li2019exploiting,
  title = {{Exploiting BERT for End-to-End Aspect-based Sentiment Analysis}},
  author = {Xin Li, Lidong Bing, Wenxuan Zhang and Wai Lam},
  booktitle = {Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-5505},
  url = {https://doi.org/10.18653/v1/d19-5505}
}

References

65 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In EMNLP, pages 452–461.
  2. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In EMNLP, pages 551–561.
  3. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In EMNLP, pages 1724–1734. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186.
  5. Finetuning BERT or Not We also study the impact of fine-tuning on the final performances. Specifically, we employ BERT to calculate the contextualized token-level representations but kept the parameters of BERT component unchanged in the training phase. Figure 3 illustrate the comparative results be… find
  6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197.
  7. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  8. Feifan Fan, Yansong Feng, and Dongyan Zhao. 2018. Multi-grained attention network for aspect-level sentiment classification. In EMNLP, pages 3433– 3442.
  9. Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In NAACL, pages 2509–2518.
  10. Zeyang Lei, Yujiu Yang, Min Yang, Wei Zhao, Jun Guo, and Yi Liu. 2019. A human-like semantic cognition network for aspect-level sentiment classification. In AAAI, pages 6650–6657. find
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR, pages 770–778. find
  12. Hao Li and Wei Lu. 2017. Learning latent sentiment scopes for entity-level sentiment analysis. In AAAI, pages 3482–3489. find
  13. Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018. Exploiting document knowledge for aspect-level sentiment classification. In ACL, pages 579–585. find
  14. Hao Li and Wei Lu. 2019. Learning explicit and implicit structures for targeted sentiment analysis. arXiv preprint arXiv:1909.07593. find
  15. Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018. Transformation networks for target-oriented sentiment classification. In ACL, pages 946–956.
  16. Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In ACL, pages 504–515.
  17. Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019a. A unified model for opinion target extraction and target sentiment prediction. In AAAI, pages 6714–6721. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  19. Zheng Li, Ying Wei, Yu Zhang, Xiang Zhang, and Xin Li. 2019b. Exploiting coarse-to-fine task transfer for aspect-level sentiment classification. In AAAI, pages 4253–4260. find
  20. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL, pages 328–339. find
  21. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR.
  22. Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. In ACL, pages 537–546.
  23. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. find
  24. Binxuan Huang and Kathleen Carley. 2018. Parameterized convolutional neural networks for aspect level sentiment classification. In EMNLP, pages 1091– 1096. find
  25. Jiangming Liu and Yue Zhang. 2017. Attention modeling for targeted sentiment. In EACL, pages 572– 577. find
  26. Binxuan Huang and Kathleen M Carley. 2019. Syntax-aware aspect level sentiment classification with graph attention networks. arXiv preprint arXiv:1909.02606.
  27. Liyuan Liu, Jingbo Shang, Xiang Ren, Frank F Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. In AAAI, pages 5253–5260.
  28. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
  29. Yang Liu. 2019. Fine-tune bert for extractive summarization. arXiv preprint arXiv:1903.10318. find
  30. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In ICML, pages 2342– 2350. find
  31. Huaishao Luo, Tianrui Li, Bing Liu, and Junbo Zhang. 2019. DOER: Dual cross-shared RNN for aspect term-polarity co-extraction. In ACL, pages 591– 601. find
  32. John D Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pages 282–289. find
  33. Dehong Ma, Sujian Li, and Houfeng Wang. 2018a. Joint learning for targeted sentiment analysis. In EMNLP, pages 4737–4742. find
  34. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL, pages 260–270.
  35. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In IJCAI, pages 4068–4074. find
  36. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  37. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In ACL, pages 1064–1074.
  38. Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. arXiv preprint arXiv:1908.11860. find
  39. Yukun Ma, Haiyun Peng, and Erik Cambria. 2018b. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm. In AAAI. find
  40. Navonil Majumder, Soujanya Poria, Alexander Gelbukh, Md. Shad Akhtar, Erik Cambria, and Asif Ekbal. 2018. IARM: Inter-aspect relation modeling with memory networks in aspect-based sentiment analysis. In EMNLP, pages 3402–3411. find
  41. Martin Schmitt, Simon Steinheber, Konrad Schreiber, and Benjamin Roth. 2018. Joint aspect and polarity classification for aspect-based sentiment analysis with end-to-end neural networks. In EMNLP, pages 1109–1114. find
  42. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NeurIPS, pages 6294– 6305.
  43. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In AAAI. find
  44. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS, pages 3111–3119.
  45. Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314. find
  46. Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In EMNLP, pages 1643–1654.
  47. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In NAACL, pages 380–385.
  48. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543.
  49. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In EMNLP, pages 214–224.
  50. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL, pages 2227–2237.
  51. Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  52. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Marı́a Jimén…
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008.
  54. Shuai Wang, Sahisnu Mazumder, Bing Liu, Mianwei Zhou, and Yi Chang. 2018. Target-sensitive memory networks for aspect sentiment classification. In ACL, pages 957–967. find
  55. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In SemEval, pages 486–495.
  56. Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In EMNLP, pages 606–615.
  57. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In SemEval, pages 27–35.
  58. Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In NAACL, pages 2324–2335.
  59. Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In ACL, pages 2514–2523. find
  60. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  61. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
  62. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  63. Jianfei Yu and Jing Jiang. 2019. Adapting bert for target-oriented multimodal sentiment classification. In IJCAI, pages 5408–5414. find
  64. Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015. Neural networks for open domain targeted sentiment. In EMNLP, pages 612–621. find
  65. Jie Zhou, Jimmy Xiangji Huang, Qin Chen, Qinmin Vivian Hu, Tingting Wang, and Liang He. 2019. Deep learning for aspect-level sentiment classification: Survey, vision and challenges. IEEE Access. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2019-exploiting-bert-end-end-aspect/