Elamur Research⌕
☰
home / papers / 2021 / Knowledge Enhanced Masked Language Model for Stance Detection

Knowledge Enhanced Masked Language Model for Stance Detection

Summary

Labeling a tweet as supporting, opposing, or neutral toward a target is difficult because posts are brief, full of new hashtags and slang, and grammatically irregular. Stance is also distinct from sentiment: earlier work found only about 60% correlation between them. The authors propose KE-MLM (Knowledge Enhanced Masked Language Modeling), a BERT fine-tuning scheme that replaces purely random masking with masking guided by task knowledge. First, a weighted log-odds-ratio with informative Dirichlet priors ranks words by how well they separate the stance classes. A background corpus serves as the prior, which dampens noise from rare words. The highest-ranked tokens are then masked preferentially during language-model fine-tuning, so attention shifts toward them before the stance classifier is trained. Experiments use a new, manually labeled Election 2020 dataset of English tweets about Joe Biden and Donald Trump. This is a low-resource setting with 2,500 annotated tweets. KE-MLM outperforms a standard fine-tuned BERT and other strong baselines. It shows that simple lexical statistics can make pretrained models more useful when labels are scarce.

Key points

  • Word selection relies on the log-odds-ratio method of Monroe et al. (2008). The authors found it a better selector of stance-relevant words than PMI, which the SKEP sentiment model used.
  • The approach adapts the sentiment-oriented SKEP idea (Tian et al., 2020) to stance and includes it as a comparison system.
  • Targets are the two 2020 US presidential candidates. The labeled set contains 2,500 tweets, and all example tweets in the paper are invented to protect user privacy.
  • The labeled stance data is publicly released at github.com/GU-DataLab/stance-detection-KE-MLM.
  • The work builds on earlier stance research using SemEval 2016 Task 6, where pretrained BERT was already a strong baseline (Ghosh et al., 2019).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

37 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 28 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.376

@inproceedings{kawintiranon2021knowledge,
  title = {{Knowledge Enhanced Masked Language Model for Stance Detection}},
  author = {Kawintiranon, Kornraphop and Singh, Lisa},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.376},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.376}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amazon. 2011. Hit review policies. https://docs.aws.amazon.com/AWSMechTurk/ latest/AWSMturkAPI/ApiReference_ HITReviewPolicies.html. ↗
  2. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics (TACL), 5:135– 146. find
  3. Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the biennial Conference of the German Society for Computational Linguistics and Language Technology (GSCL). find
  4. Ceren Budak. 2019. What happened? the spread of fake news publisher content during the 2016 us presidential election. In Proceedings of the World Wide Web Conference (WWW). find
  5. Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  6. Kevin Crowston. 2012. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Shaping the Future of ICT Research. Methods and Approaches. find
  7. Marco Del Tredici, Diego Marcheggiani, Sabine Schulte im Walde, and Raquel Fernández. 2019. You shall know a user by the company it keeps: Dynamic representations for social media users in NLP. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th Internati… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  9. Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2017. Twitter stance detection—a subjectivity and sentiment polarity inspired two-phase approach. In IEEE international conference on data mining workshops (ICDMW). find
  10. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI). find
  11. Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: a comparative study. Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 75–87. find
  12. Marco Guerini, Lorenzo Gatti, and Marco Turchi. 2013. Sentiment analysis: How to derive prior polarities from SentiWordNet. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  14. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  15. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics (TACL), 8:64–77. find our card
  16. Dan Jurafsky, Victor Chahuneau, Bryan R Routledge, and Noah A Smith. 2014. Narrative framing of consumer sentiment in online restaurant reviews. First Monday. find
  17. Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. SentiLARE: Sentiment-aware language representation learning with linguistic knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  18. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). find
  19. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1). find
  20. Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  22. Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  23. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find
  24. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find
  25. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  26. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  27. Lahari Poddar, Wynne Hsu, and Mong Li Lee. 2017. Author-aware aspect topic sentiment model to retrieve supporting opinions from reviews. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. find
  29. Martin Schmitt, Simon Steinheber, Konrad Schreiber, and Benjamin Roth. 2018. Joint aspect and polarity classification for aspect-based sentiment analysis with end-to-end neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  30. Anirban Sen, Manjira Sinha, Sandya Mannarswamy, and Shourya Roy. 2018. Stance classification of multi-perspective consumer health information. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data. find
  31. Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019. Tweet stance detection using an attention based neural ensemble model. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  32. Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  33. Rui Wang, Deyu Zhou, Mingmin Jiang, Jiasheng Si, and Yang Yang. 2019. A survey on opinion mining: From stance to product aspect. IEEE Access, 7:41101–41124. find
  34. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at SemEval-2016 task 6 : A specific convolutional neural network system for effective stance detection. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find our card
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  36. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  37. Guido Zarrella and Amy Marsh. 2016. Mitre at semeval-2016 task 6: Transfer learning for stance detection. In Proceedings of the International Workshop on Semantic Evaluation (SemEval). find our card
  38. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  39. Jie Zhou, Junfeng Tian, Rui Wang, Yuanbin Wu, Wenming Xiao, and Liang He. 2020. SentiX: A sentiment-aware pre-trained model for cross-domain sentiment analysis. In Proceedings of the International Conference on Computational Linguistics (COLING). find
  40. Parinaz Sobhani, Saif Mohammad, and Svetlana Kiritchenko. 2016. Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM). find
  41. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kawintiranon-2021-knowledge-enhanced-masked-language-model/