Elamur Research⌕
☰
home / papers / 2023 / Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media

Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media

Paper · 2023 · Bowen Zhang, Xianghua Fu, Daijun Ding et al.

Summary

This paper asks whether a very large language model can perform stance detection on social media without any training. Stance detection means labelling a tweet as favouring, opposing, or neutral toward a given target. Fine-tuning models with hundreds of billions of parameters is too expensive, so the authors test chain-of-thought (CoT) prompting with ChatGPT (GPT-3.5). They compare two strategies. DQA simply asks the model for the stance. StSQA first has the model explain one worked example, then supplies that question–answer pair (QAP) as a one-shot demonstration. The evaluation covers SemEval-2016, VAST and P-Stance. Prompted ChatGPT is compared with trained baselines in both zero-shot and in-domain settings, where the baselines learn from 80% of the data. StSQA performs best on most SemEval-2016 and P-Stance targets, but it trails the strongest baselines in some cases. The authors also identify three limitations: the model's own stance bias, sensitivity to how demonstrations are chosen, and targets that have several facets. The work matters because it shows training-free prompting can be a practical option for stance detection, and it marks out open problems.

Key points

  • Datasets: SemEval-2016 has 4,870 tweets across 6 targets. VAST is a zero-shot benchmark with 4,003 training samples. P-Stance has 21,574 political tweets about Donald Trump, Joe Biden and Bernie Sanders. The metrics are F avg and macro-F1 (F m).
  • Zero-shot on SemEval-2016: StSQA reaches F avg of 78.9 (HC), 68.7 (FM), 61.5 (LA) and 71.6 (DT). The authors report an average gain of about 16.6% over PT-HCL. On VAST, however, StSQA scores 68.9 F m, below PT-HCL at 71.6.
  • In-domain on P-Stance: StSQA beats Bert_Spc on every target, with F avg of 85.7 (Trump), 82.8 (Biden) and 80.8 (Bernie). On SemEval-2016, it falls well short of TPDG on the LA target (61.8 vs 74.7 F m).
  • Stance bias: ChatGPT appears to support abortion legalization by default. As a result, it labels many neutral LA tweets as favourable, and the authors report a similar effect on VAST.
  • Demonstrations and task framing: Adding more QAPs (tested from 1 to 5) can confuse the model's reasoning. Explanations at the semantic level work better than word-level ones. The authors also argue for fine-grained stance definitions for multi-faceted targets, for example Trump viewed as an actor versus as a politician.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China152 works · 114,627 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates145 works · 6,811 citations🎓Shenzhen Technology UniversityShenzhen · China5 works · 98 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksSentiment & emotion · 1,037 works

Cite

Cite as Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Genan Dai, Nan Yin, Yangyang Li and Liwen Jing. 2023. Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media. arXiv:2304.03087.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.03087 · arXiv 2304.03087

@misc{zhang2023investigating,
  title = {{Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media}},
  author = {Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Genan Dai, Nan Yin, Yangyang Li and Liwen Jing},
  year = {2023},
  doi = {10.48550/arxiv.2304.03087},
  eprint = {2304.03087},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.03087}
}

References

24 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597.
  2. Emily Allaway and Kathleen Mckeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931.
  3. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 1–9.
  4. I Augenstein, T Rocktaeschel, A Vlachos, and K Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Sheffield.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186. Association for Computational Linguistics.
  6. Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical stance detection for twitter: A two-phase lstm model using attention. In European Conference on Information Retrieval, pages 529–536. Springer. find
  7. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. International Joint Conferences on Artificial Intelligence. find
  8. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. arXiv preprint arXiv:2212.04092. find
  9. Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, pages 197–… find
  10. Xinyu Jiang, Bowen Zhang, Yunming Ye, and Zhenhua Liu. 2019. A hierarchical model with recurrent convolutional neural networks for sequential sentence classification. In Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 20… find
  11. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1– 37. find
  12. Ang Li, Bin Liang, Jingqian Zhao, Bowen Zhang, Min Yang, and Ruifeng Xu. 2023. Stance detection on social media with background knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15703–15717.
  13. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365.
  14. Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2738–2747. find
  15. Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu. 2021. BertGCN: Transductive text classification by combining GNN and BERT. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1456–1462, Online. Association for Computational Lingui… find
  16. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California.
  17. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  18. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4,.
  19. Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 592–596.
  20. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  21. Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778– 783.
  22. Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3188–3197.
  23. Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In EMNLP/IJCNLP (1).
  24. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-investigating-chain-thought-chatgpt-stance/