Elamur Research⌕
☰
home / papers / 2023 / Chain-of-Thought Embeddings for Stance Detection on Social Media

Chain-of-Thought Embeddings for Stance Detection on Social Media

Summary

Social media posts often express stance indirectly, through slang, allusions, or emerging events that an LLM with a fixed knowledge cut-off may not grasp. The authors show that Chain-of-Thought (COT) prompting with ChatGPT still fails in two ways. In Implicit Stance Confusion, the model misses an indirect reference to the target. In Stance Label Hallucination, the reasoning is sound but the label is attached to the wrong entity or formatted badly.

Their remedy, COT Embeddings, treats the generated reasoning as input text rather than as a final answer. A Twitter-pretrained RoBERTa is fine-tuned on the COT text, either alone or paired with the tweet. The encoder can then:

  • use correct reasoning even when the parsed label is wrong,
  • rely on patterns learned from training data when the LLM is confused,
  • absorb world knowledge contained in the reasoning.

On Tweet-Stance and Presidential-Stance, the method sets new state-of-the-art scores. This suggests LLM rationales are more valuable as features for a supervised classifier than as direct predictions.

Key points

  • Setup: ChatGPT (gpt-3.5) produces 1-shot COT reasoning with a bracketed label. The encoder is cardiffnlp/twitter-roberta-base-sep2022, tested in three variants: TR-Tweet, TR-COT and TR-Tweet+COT (pair input).
  • Tweet-Stance (5 topics, macro F1 over Favor/Against): TR-Tweet+COT reaches 76.3 F1, 6.1 points above ChatGPT COT alone. TR-COT alone reaches 75.7.
  • Presidential-Stance (Biden/Trump, averaged over 3 seeds):
    • On Biden, TR-Tweet+COT scores 77.1 macro F1, beating KE-MLM (75.7). TR-COT improves on raw ChatGPT COT by 20.7 points.
    • On Trump, TR-COT, which never sees the tweet, achieves the best score: 81.5 versus 78.7 for KE-MLM.
  • Error analysis: Of the Tweet-Stance cases the model fixed relative to ChatGPT, 74% (131/175) were tweets that ChatGPT had wrongly labelled neutral.
  • LLM quality matters: COT from Llama-2-7b-chat helps on some subsets, while COT from Falcon-7b-instruct is largely unhelpful. The authors also note limitations: sensitivity to the prompt, possibly weaker gains outside fast-changing social media language, and extra inference cost if COT cannot be precomputed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.273 · arXiv 2310.19750

@inproceedings{gatto2023chain,
  title = {{Chain-of-Thought Embeddings for Stance Detection on Social Media}},
  author = {Gatto, Joseph and Sharif, Omar and Preum, Sarah M.},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.273},
  eprint = {2310.19750},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.273}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 7b, only provides a minor improvement over the baseline TR-Tweet. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes-low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Pe… find
  2. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. Association for Computational… find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  4. Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance detection for mis- and disinformation identification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1259–1277, Seattle, United States. Association for Computational Lingu… find our card
  5. Zihao He, Negar Mokhberian, and Kristina Lerman. 2022. Infusing knowledge from wikipedia to enhance stance detection. find our card
  6. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4725–4735, Online. Association for Com… find our card
  7. Yingjie Li and Cornelia Caragea. 2021. Target-aware data augmentation for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850–1860, Online. Association for Computational Lingui… find our card
  8. Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 2738–2747, New York, NY, USA. Association for Computing Machinery. find
  9. Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu. 2021. BertGCN: Transductive text classification by combining GNN and BERT. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1456–1462, Online. Association for Computational Lingui… find
  10. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3152–3157, Online. Association for Computational Linguistics. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  12. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41. find
  13. Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4067–4076, Online. Association for Com… find
  14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  15. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. find our card
  16. Bowen Zhang, Daijun Ding, and Liwen Jing. 2023a. How would stance detection techniques evolve after the launch of chatgpt? find
  17. Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023b. Investigating chain-of-thought with chatgpt for stance detection on social media. find
  18. Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021. Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph. In find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gatto-2023-chain-thought-embeddings-stance-detection/