Elamur Research⌕
☰
home / papers / 2020 / SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Paper · 2020 · Hao Tian, Can Gao, Xinyan Xiao et al.

Summary

Pre-trained language models such as BERT and RoBERTa learn general representations through word-level objectives. They never explicitly model the sentiment cues that classical sentiment analysis depends on: sentiment words, their polarity, and aspect–sentiment pairs. SKEP, developed at Baidu and USTC, builds this knowledge into pre-training so that a single representation can serve several sentiment tasks.

The knowledge is mined automatically from unlabeled text. A PMI-based method following Turney (2002) assigns polarity to candidate words, starting from a small set of seed words. Each sentiment word is then linked to a nearby noun to form aspect–sentiment pairs. SKEP replaces random masking with masking of these detected items, and a transformer encoder learns to restore them through three jointly optimized objectives.

The model was tested on sentence-level classification, aspect-level classification and opinion extraction. SKEP clearly outperforms the RoBERTa baseline and reaches new state-of-the-art results on most test sets. This shows that sentiment-aware pre-training is worth the effort.

Key points

  • Three pre-training objectives:
    • Sentiment Word (SW) prediction.
    • Word Polarity (SP) prediction.
    • Aspect-Sentiment pair (AP) prediction, cast as multi-label classification from the final [CLS] state so the model captures how aspect and sentiment words depend on each other.
  • Knowledge mining: sentiment word polarity is the difference between PMI scores with positive and negative seeds. An aspect noun must lie within 3 tokens of its sentiment word.
  • Hybrid masking:
    • At most 2 aspect–sentiment pairs are masked as whole units.
    • Masked sentiment words are capped below 10% of tokens.
    • Any remaining budget is filled with RoBERTa-style random token masking.
  • Pair representation ablation: predicting pairs from [CLS] (Sent-Vector) and from concatenated word vectors (Pair-Vector) gave nearly identical results. [CLS] was kept because it is more efficient.
  • Attention visualization and code: on SST-2 and Sem-L examples, SKEP attends to sentiment words such as "amazing" that RoBERTa overlooks, suggesting better interpretability. The code is available at github.com/baidu/Senta.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China40 works · 14,627 citations🎓University of Science and Technology of ChinaHefei · China26 works · 111,012 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,344 worksSentiment & emotion · 609 works

Cite

Cite as Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang and Feng Wu. 2020. SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.374

@inproceedings{tian2020skep,
  title = {{SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis}},
  author = {Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang and Feng Wu},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.374},
  url = {https://doi.org/10.18653/v1/2020.acl-main.374}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the Workshop on Language in Social Media (LSM 2011).
  2. Jeremy Barnes, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2019. Lexicon information in neural sentiment analysis: a multi-task learning approach. In Proceedings of the 22nd Nordic Conference on Computational Linguistics. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL 2019.
  4. Ying Ding, Jianfei Yu, and Jing Jiang. 2017. Recurrent neural networks with auxiliary labels for cross-domain opinion target extraction. In AAAI 2017. find
  5. Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Lidong Bing, Min Yang, Ruifeng Xu, and Ruibin Mao. 2019. A knowledge regularized hierarchical approach for emotion cause analysis. In EMNLP 2019. find
  6. Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017. A question answering approach for emotion cause extraction. In EMNLP 2017. find
  7. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In EMNLP 2019.
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  9. Ana Marasović and Anette Frank. 2018. SRL4ORL: Improving opinion role labeling using multi-task learning with semantic role labeling. In NAACL 2018. find
  10. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. find
  11. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
  12. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In SemEval 2014.
  13. Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2017. Linguistically regularized LSTM for sentiment classification. In ACL 2017. find
  14. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. find
  15. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In ACL 2019.
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer.
  17. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
  18. Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. ArXiv, abs/1908.11860. find
  19. Bonggun Shin, Timothy Lee, and Jinho D. Choi. 2017. Lexicon integrated CNN models with attention for sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. find
  20. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP 2013. find
  21. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
  22. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307.
  23. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL 2014.
  24. Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In ACL 2002.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017.
  26. Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In IJCAI 2015. find
  27. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
  28. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. ArXiv, abs/1703.06345. find
  29. Ziqian Zeng, Wenxuan Zhou, Xin Liu, and Yangqiu Song. 2019. A variational approach to weakly supervised document-level multi-aspect sentiment classification. In NAACL 2019. find
  30. Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis : A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. find
  31. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS 2015.
  32. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In ACL 2019. find
  33. Pinlong Zhao, Linlin Hou, and Ou Wu. 2019. Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification. CoRR, abs/1906.04501. — [arXiv:1906.04501](https://arxiv.org/abs/1906.04501) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2020-skep-sentiment-knowledge-enhanced-pre/