Elamur Research⌕
☰
home / papers / 2022 / How would Stance Detection Techniques Evolve after the Launch of ChatGPT?

How would Stance Detection Techniques Evolve after the Launch of ChatGPT?

Paper · 2022 · Bowen Zhang, Daijun Ding, Liwen Jing et al.

Summary

This short paper asks whether ChatGPT changes how stance detection is done. Stance detection means labelling a text's position toward a target as favor, against, or neutral. The authors test zero-shot prompting: each tweet and target go into one direct question that asks ChatGPT to choose a stance. No training is involved. The latest version reports results for GPT-3.5-0301. Evaluation uses two English Twitter benchmarks, SemEval-2016 and P-Stance, scored with F1-avg and macro-F1. ChatGPT is compared with zero-shot baselines and with supervised models trained on 80% of the data. In the zero-shot setting, ChatGPT clearly beats all earlier systems. Against supervised models it wins on most targets but not all. The authors also show that ChatGPT can justify its labels in natural language. It can even say why a tweet cannot be classified, which helps flag poorly annotated examples. They suggest that prompting large dialogue models could reshape the field. The evidence is limited, however: a single prompt template, two datasets, and no API access at the time of the experiments.

Key points

  • Data:
    • SemEval-2016 has 4,870 tweets. The paper uses three of its six targets: Hillary Clinton, Feminist Movement and Legalization of Abortion.
    • P-Stance has 21,574 political tweets about Donald Trump, Joe Biden and Bernie Sanders. Only favor and against labels are scored.
  • Zero-shot results (SemEval-2016, F1-avg):
    • ChatGPT scores 78.0 on HC, 69.0 on FM and 59.3 on LA.
    • The strongest baseline, PT-HCL, scores 54.5, 54.6 and 50.9.
  • In-domain results (SemEval-2016):
    • ChatGPT leads on HC (79.5 F1-m vs. 73.4 for TPDG).
    • It narrowly leads on FM (68.4 vs. 67.3).
    • It falls well behind on LA (58.2 F1-m vs. 74.7 for TPDG).
  • In-domain results (P-Stance):
    • ChatGPT slightly outperforms Bert_Spc on all three targets.
    • Its F1-m scores are 82.8 for Trump, 82.3 for Biden and 79.4 for Sanders. Bert_Spc scores 81.4, 81.5 and 78.3.
  • Proposed future directions:
    • Better prompt engineering.
    • Explainable stance classification.
    • Multi-turn dialogue that adds background knowledge or examples to resolve tweets that are ambiguous or missing context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai and Nan Yin. 2022. How would Stance Detection Techniques Evolve after the Launch of ChatGPT?. arXiv:2212.14548.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.14548 · arXiv 2212.14548

@misc{zhang2022how,
  title = {{How would Stance Detection Techniques Evolve after the Launch of ChatGPT?}},
  author = {Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai and Nan Yin},
  year = {2022},
  doi = {10.48550/arxiv.2212.14548},
  eprint = {2212.14548},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.14548}
}

References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aseel Addawood, Jodi Schneider, and Masooda Bashir. 2017. Stance classification of twitter debates: The encryption debate as a use case. In Proceedings of the 8th international conference on Social Media & Society, pages 1–10. find
  2. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597.
  3. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 1–9.
  4. I Augenstein, T Rocktaeschel, A Vlachos, and K Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Sheffield.
  5. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2021. Synthetic examples improve cross-target generalization: A study on stance detection on a twitter corpus. In Proceedings of the Eleventh Workshop on Computational Approaches to Su…
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186. Association for Computational Linguistics.
  7. Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical stance detection for twitter: A two-phase lstm model using attention. In European Conference on Information Retrieval, pages 529–536. Springer. find
  8. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. International Joint Conferences on Artificial Intelligence. find
  9. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies. ACL.
  10. Yaakov HaCohen-Kerner, Ziv Ido, and Ronen Ya’akobov. 2017. Stance classification of tweets using skip char ngrams. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 266–278. Springer. find
  11. Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, pages 197–… find
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  13. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1– 37. find
  14. Chen Li, Hao Peng, Jianxin Li, Lichao Sun, Lingjuan Lyu, Lihong Wang, Philip S. Yu, and Lifang He. 2022. Joint stance and rumor detection in hierarchical heterogeneous graph. IEEE Trans. Neural Networks Learn. Syst., 33(6):2530–2542. find
  15. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365.
  16. Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2738–2747. find
  17. Bin Liang, Yonghao Fu, Lin Gui, Min Yang, Jiachen Du, Yulan He, and Ruifeng Xu. 2021. Target-adaptive graph for cross-target stance detection. In WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, pages 3453–3464. find
  18. Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu. 2021. BertGCN: Transductive text classification by combining GNN and BERT. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1456–1462, Online. Association for Computational Lingui… find
  19. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
  20. Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021b. Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3152–3157.
  21. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California.
  22. Saif M Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3):1–23.
  23. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  24. Vasiliki Simaki, Carita Paradis, and Andreas Kerren. 2017. Stance classification in texts from blogs on the 2016 british referendum. In International Conference on Speech and Computer, pages 700–709. Springer. find
  25. Maria Skeppstedt, Vasiliki Simaki, Carita Paradis, and Andreas Kerren. 2017. Detection of stance and sentiment modifiers in political blogs. In International Conference on Speech and Computer, pages 302–311. Springer. find
  26. Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierarchical attention network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2399–2409.
  27. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4,.
  28. Adam Tsakalidis, Nikolaos Aletras, Alexandra I Cristea, and Maria Liakata. 2018. Nowcasting the stance of social media users in a sudden vote: The case of the greek referendum. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 367–376. find
  29. Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 592–596.
  30. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  31. Michael Wojatzki and Torsten Zesch. 2016. Stance-based argument mining–modeling implicit argumentation using stance. In Proceedings of the KONVENS, pages 313–322. find
  32. Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778– 783.
  33. Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020a. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. Association for Computational Linguistics.
  34. Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020b. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3188–3197.
  35. Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In EMNLP/IJCNLP (1).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2022-would-stance-detection-techniques-evolve/