Elamur Research⌕
☰
home / papers / 2023 / Stance Detection with Collaborative Role-Infused LLM-Based Agents

Stance Detection with Collaborative Role-Infused LLM-Based Agents

Paper · 2023 · Xiaochong Lan, Chen Gao, Depeng Jin et al.

Summary

Stance detection identifies whether a text's author favors, opposes, or is neutral toward a given target. Most existing methods, including cross-target and zero-shot ones, still need labeled training data. Simple prompting of LLMs often falls short of strong non-LLM baselines for two reasons: understanding the text requires knowledge from several areas, and stances are usually implied rather than stated.

The authors propose COLA, a training-free pipeline of role-playing LLM agents with GPT-3.5 Turbo as the main backbone. It runs in three stages:

  1. Analysis. Three agents examine the text separately, each from its own angle: one specializing in language, one in subject-matter knowledge, and one seasoned in social media.
  2. Debate. One agent per stance label builds an argument from these analyses.
  3. Decision. A final judge weighs the debate together with the original text.

COLA outperforms all other zero-shot methods on SEM16, P-Stance and VAST. It roughly matches in-target supervised systems, and ablations show every module helps. The framework also produces readable explanations and transfers to other classification tasks. This matters for social-media analysis where annotated data for new targets is scarce.

Key points

  • On SEM16, COLA beats the best non-LLM zero-shot stance baseline by 19.2 absolute points.
  • It needs no annotated data or fine-tuning and works through API calls. Gains were also reported with open-source LLMs.
  • The analyst–debater–summarizer design was reused for aspect-based sentiment analysis. It was also applied to persuasion prediction on the Wang et al. (2019) dataset, where it beat Hybrid RCNN and plain GPT-3.5 Turbo on Accuracy and Macro-F1.
  • A noted limitation is weaker handling of current events. The authors plan to add a real-time knowledge base to the analysis stage.
  • The paper was published at AAAI 2024. SEM16 and P-Stance tweets were obtained via the Twitter API using Tweet IDs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China439 works · 94,783 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksGeneral Social Sciences · 256 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,037 works

Cite

Cite as Xiaochong Lan, Chen Gao, Depeng Jin and Yong Li. 2023. Stance Detection with Collaborative Role-Infused LLM-Based Agents. arXiv:2310.10467.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.10467 · arXiv 2310.10467

@misc{lan2023stance,
  title = {{Stance Detection with Collaborative Role-Infused LLM-Based Agents}},
  author = {Xiaochong Lan, Chen Gao, Depeng Jin and Yong Li},
  year = {2023},
  doi = {10.48550/arxiv.2310.10467},
  eprint = {2310.10467},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10467}
}

References

49 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AlDayel, A.; and Magdy, W. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4): 102597.
  2. Allaway, E.; and McKeown, K. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. arXiv preprint arXiv:2010.03640.
  3. Allaway, E.; Srikanth, M.; and McKeown, K. 2021. Adversarial learning for zero-shot stance detection on social media. arXiv preprint arXiv:2105.06603. Arrieta, A. B.; Dı́az-Rodrı́guez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; Garcı́a, S.; Gil-López, S.; Molina, D.; Benjamins, R.; et a…
  4. Bar-Haim, R.; Bhattacharya, I.; Dinuzzo, F.; Saha, A.; and Slonim, N. 2017. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 251–261.
  5. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877– 1901.
  6. Cai, T.; Wang, X.; Ma, T.; Chen, X.; and Zhou, D. 2023. Large language models as tool makers. arXiv preprint arXiv:2305.17126.
  7. Chan, C.-M.; Chen, W.; Su, Y.; Yu, J.; Xue, W.; Zhang, S.; Fu, J.; and Liu, Z. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. arXiv preprint arXiv:2308.07201.
  8. Chen, C.; Teng, Z.; Wang, Z.; and Zhang, Y. 2022. Discrete opinion tree induction for aspect-based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2051–2064. find
  9. Chen, P.; Sun, Z.; Bing, L.; and Yang, W. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of the 2017 conference on empirical methods in natural language processing, 452–461.
  10. Das, A.; and Rad, P. 2020. Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv preprint arXiv:2006.11371. — [arXiv:2006.11371](https://arxiv.org/abs/2006.11371) find
  11. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  12. Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv preprint arXiv:2305.14325.
  13. Gao, C.; Lan, X.; Li, N.; Yuan, Y.; Ding, J.; Zhou, Z.; Xu, F.; and Li, Y. 2023a. Large language models empowered agent-based modeling and simulation: A survey and perspectives. arXiv preprint arXiv:2312.11970.
  14. Gao, C.; Lan, X.; Lu, Z.; Mao, J.; Piao, J.; Wang, H.; Jin, D.; and Li, Y. 2023b. S 3 : Social-network Simulation System with Large Language Model-Empowered Agents. arXiv preprint arXiv:2307.14984. Grčar, M.; Cherepnalkoski, D.; Mozetič, I.; and Kralj Novak, P. 2017. Stance and influence of Twitte… — [arXiv:2307.14984](https://arxiv.org/abs/2307.14984) find
  15. Hong, S.; Zheng, X.; Chen, J.; Cheng, Y.; Zhang, C.; Wang, Z.; Yau, S. K. S.; Lin, Z.; Zhou, L.; Ran, C.; et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352.
  16. Huang, H.; Zhang, B.; Li, Y.; Zhang, B.; Sun, Y.; Luo, C.; and Peng, C. 2023. Knowledge-enhanced prompt-tuning for stance detection. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(6): 1–20. find
  17. Jang, M.; and Allan, J. 2018. Explaining controversy on social media via stance summarization. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 1221–1224. find
  18. Khiabani, P. J.; and Zubiaga, A. 2024. SocialPET: Socially Informed Pattern Exploiting Training for Few-Shot Stance Detection in Social Media. arXiv preprint arXiv:2403.05216. Küçük, D.; and Can, F. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1): 1–37. — [arXiv:2403.05216](https://arxiv.org/abs/2403.05216) find
  19. Li, A.; Liang, B.; Zhao, J.; Zhang, B.; Yang, M.; and Xu, R. 2023a. Stance Detection on Social Media with Background Knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 15703– 15717.
  20. Li, N.; Gao, C.; Li, Y.; and Liao, Q. 2023b. Large language model-empowered agents for simulating macroeconomic activities. arXiv preprint arXiv:2310.10436. — [arXiv:2310.10436](https://arxiv.org/abs/2310.10436) find
  21. Li, Y.; Sosea, T.; Sawant, A.; Nair, A. J.; Inkpen, D.; and Caragea, C. 2021. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2355– 2365.
  22. Liang, B.; Chen, Z.; Gui, L.; He, Y.; Yang, M.; and Xu, R. 2022a. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, 2738– 2747. find
  23. Liang, B.; Fu, Y.; Gui, L.; Yang, M.; Du, J.; He, Y.; and Xu, R. 2021. Target-adaptive graph for cross-target stance detection. In Proceedings of the Web Conference 2021, 3453– 3464. find
  24. Liang, B.; Zhu, Q.; Li, X.; Yang, M.; Gui, L.; He, Y.; and Xu, R. 2022b. Jointcl: a joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, 81–91. Association fo…
  25. Liang, T.; He, Z.; Jiao, W.; Wang, X.; Wang, Y.; Wang, R.; Yang, Y.; Tu, Z.; and Shi, S. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. arXiv preprint arXiv:2305.19118.
  26. Liu, R.; Lin, Z.; Tan, Y.; and Wang, W. 2021. Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 3152–3157.
  27. Lozhnikov, N.; Derczynski, L.; and Mazzara, M. 2020. Stance prediction for russian: data and analysis. In Proceedings of 6th International Conference in Software Engineering for Defence Applications: SEDA 2018 6, 176–186. find
  28. Springer. Mohammad, S.; Kiritchenko, S.; Sobhani, P.; Zhu, X.; and Cherry, C. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), 31–41.
  29. Nair, V.; Schumacher, E.; Tso, G.; and Kannan, A. 2023. DERA: enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303.17071.
  30. OpenAI, R. 2023. GPT-4 technical report. arXiv, 2303– 08774.
  31. Park, J. S.; O’Brien, J. C.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. find
  32. Pontiki, M.; Galanis, D.; Papageorgiou, H.; Androutsopoulos, I.; Manandhar, S.; AL-Smadi, M.; Al-Ayyoub, M.; Zhao, Y.; Qin, B.; De Clercq, O.; et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In ProWorkshop on Semantic Evaluation (SemEval-2016), 19–30. Association for Computational…
  33. Pontiki, M.; Galanis, D.; Pavlopoulos, J.; Papageorgiou, H.; Androutsopoulos, I.; and Manandhar, S. 2014. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 27–35. Dublin, Ireland: Association for Computationa…
  34. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. Schick, T.; Dwivedi-Yu, J.; Dessı̀, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.;…
  35. Shinn, N.; Cassano, F.; Labash, B.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning (arXiv: 2303.11366). arXiv.
  36. Tang, H.; Ji, D.; Li, C.; and Zhou, Q. 2020. Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification. In Proceedings of the 58th annual meeting of the association for computational linguistics, 6578–6588.
  37. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  38. Wang, X.; Shi, W.; Kim, R.; Oh, Y.; Yang, S.; Zhang, J.; and Yu, Z. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725.
  39. Wang, Y.; Huang, M.; Zhu, X.; and Zhao, L. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, 606–615.
  40. Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  41. Wei, P.; and Mao, W. 2019. Modeling transferable topics for cross-target stance detection. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 1173–1176. find
  42. Wei, P.; Mao, W.; and Zeng, D. 2018. A target-guided neural memory model for stance detection in twitter. In 2018 International Joint Conference on Neural Networks (IJCNN), 1–8. find
  43. IEEE. Xiang, J.; Tao, T.; Gu, Y.; Shu, T.; Wang, Z.; Yang, Z.; and Hu, Z. 2023. Language Models Meet World Models: Embodied Experiences Enhance Language Models. arXiv preprint arXiv:2305.10626. find
  44. Xu, C.; Paris, C.; Nepal, S.; and Sparks, R. 2018. Cross-target stance classification with self-attention networks. arXiv preprint arXiv:1805.06593.
  45. Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  46. Zhang, B.; Ding, D.; and Jing, L. 2022. How would stance detection techniques evolve after the launch of chatgpt? arXiv preprint arXiv:2212.14548.
  47. Zhang, B.; Fu, X.; Ding, D.; Huang, H.; Li, Y.; and Jing, L. 2023. Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media. arXiv preprint arXiv:2304.03087.
  48. Zhang, C.; Li, Q.; and Song, D. 2019. Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJ…
  49. Ziems, C.; Held, W.; Shaikh, O.; Chen, J.; Zhang, Z.; and Yang, D. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lan-2023-stance-detection-collaborative-role-infused/