Elamur Research⌕
☰
home / papers / 2025 / Can Large Language Models Address Open-Target Stance Detection?

Can Large Language Models Address Open-Target Stance Detection?

Summary

The paper examines stance detection in the most realistic scenario, which the authors call Open-Target Stance Detection (OTSD). Here, the target toward which the text's author expresses a stance does not occur in the training data and is not given to the model. The model must formulate the target from the text itself and then determine the stance toward it: favor, against, or none. The only prior approach, TSE, matches generated targets against a predefined list, but in OTSD no such list exists. The authors tested eight LLMs from the GPT, Gemini, Llama, and Mistral families in a zero-shot setting with "Task Definition"-style prompts. They compared two variants: a two-step one (TG+SD) and a joint one-step one (TG&SD). Evaluation was conducted on the TSE, VAST, and EZSTANCE corpora. To evaluate generated targets, the authors propose the BTSD metric, based on a fine-tuned BERTweet, which agrees well with human judgments. LLMs outperformed TSE in both target generation and stance detection, and the one-step prompt generally yielded better results. However, when the target is not explicitly named in the text, performance drops noticeably. The value of the paper lies in posing a more practical task and providing a tool for evaluating it.

Key points

  • Data: TSE (3000 examples, 6 unique targets), VAST (5100 examples, 2145 targets), and EZSTANCE (9313 examples, 6873 targets). The multi-target VAST and EZSTANCE were converted to a single-target format. Examples were split into explicit and non-explicit depending on whether the target is mentioned in the text after lemmatization.
  • Models: GPT-3.5, GPT-4o, Gemini-flash-8B, Gemini-pro, Llama-3-8B, Llama-3-70B, Mistral-small, Mistral-large. Among the open models, the strongest are Mistral-large and Llama-3-70B, which outperform each other depending on the setting.
  • The BTSD metric is BERTweet fine-tuned on SemEval, AM, COVID-19, and P-Stance (19 targets in total). The macro-F1 of this classifier serves as a measure of the quality of generated targets. The Kendall correlation with human judgments is τ = 0.85 / 0.74 / 0.79 for TSE / VAST / EZSTANCE. For comparison, semantic similarity on BERT embeddings (SemSim) yields only about 0.57–0.59.
  • Human evaluation: 500 random examples from each corpus, 3 annotators, a 0 / 0.5 / 1 scale. Agreement: Krippendorff α = 0.76, Fleiss κ = 0.664.
  • A typical error: the model may generate a target that is close in topic but opposite in meaning, and the stance is then inverted. For example, GPT-4o produced "gun control" where the reference target is "permit to carry gun". The authors suggest LLM-as-a-judge as a possible way to evaluate target–stance consistency. The data and code are released on GitHub (AbuUbaida/opentarget), but the work covers only English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.54 · arXiv 2409.00222

@inproceedings{akash2025can,
  title = {{Can Large Language Models Address Open-Target Stance Detection?}},
  author = {Akash, Abu Ubaida and Fahmy, Ahmed and Trabelsi, Amine},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.54},
  eprint = {2409.00222},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.54}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Emily Allaway and Kathleen McKeown. 2020. Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931, Online. Association for Computational Linguistics. find our card
  3. Emily Allaway, Malavika Srikanth, and Kathleen McKeown. 2021. Adversarial learning for zero-shot stance detection on social media. arXiv preprint arXiv:2105.06603. find our card
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  5. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  6. Akash Gautam, Puneet Mathur, Rakesh Gosangi, Debanjan Mahata, Ramit Sawhney, and Rajiv Ratn Shah. 2020. \# metooma: Multi-aspect annotations of tweets related to the metoo movement. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 209–216. find
  7. Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detection in covid-19 tweets. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (long papers)… find our card
  8. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  9. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. find
  10. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  11. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1– 37. find
  12. Yingjie Li and Cornelia Caragea. 2021. A multi-task learning framework for multi-target stance detection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2320–2326. find our card
  13. Yingjie Li, Krishna Garg, and Cornelia Caragea. 2023a. A new direction in stance detection: Target-stance extraction in the wild. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10071– 10085. find our card
  14. Yingjie Li, Chenye Zhao, and Cornelia Caragea. 2023b. Tts: A target-based teacher-student framework for zero-shot stance detection. In Proceedings of the ACM Web Conference 2023, pages 1500–1509. find
  15. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won’t-they: A very large dataset for stance detection on twitter. arXiv preprint arXiv:2005.00388. find our card
  16. Iain J Cruickshank and Lynnette Hui Xian Ng. 2023. Use of large language models for stance classification. arXiv preprint arXiv:2309.13734. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  18. Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2738–2747. find
  19. Yun Luo, Zihan Liu, Yuefeng Shi, Stan Z Li, and Yue Zhang. 2022. Exploiting sentiment and common sense for zero-shot stance detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 7112–7123. find our card
  20. Lin Miao, Mark Last, and Marina Litvak. 2020. Twitter data augmentation for monitoring public opinion on covid-19 intervention measures. In Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. find
  21. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3945– 3952. find
  22. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41. find
  23. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. arXiv preprint arXiv:2005.10200. find our card
  24. Dexter Adamson Njuka and Jackson Phiri. 2021. Factors influencing social media in managing corporate reputation for a christian organisation in developing countries based on the vt4 model. Technology and Investment, 12(2):66–81. find
  25. Amit Singhal. 2001. Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24:35–43. find
  26. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, pages 116–124. find
  27. Christian Stab, Tristan Miller, and Iryna Gurevych. 2018. Cross-topic argument mining from heterogeneous sources using attention-based neural networks. arXiv preprint arXiv:1802.05758. find
  28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  29. Haoyang Wen and Alexander G Hauptmann. 2023. Zero-shot and few-shot stance detection on varied topics via conditional generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1491–1499. find our card
  30. Hanzi Xu, Slobodan Vucetic, and Wenpeng Yin. 2022. Openstance: Real-world zero-shot stance detection. arXiv preprint arXiv:2210.14299. find our card
  31. Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3188–3197. find our card
  32. Jiarui Zhang, Shaojuan Wu, Xiaowang Zhang, and Zhiyong Feng. 2023. Task-specific data augmentation for zero-shot and few-shot stance detection. In Companion Proceedings of the ACM Web Conference 2023, pages 160–163. find
  33. Chenye Zhao and Cornelia Caragea. 2024. EZ-STANCE: A large dataset for English zero-shot stance detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15697– 15714, Bangkok, Thailand. Association for Computational Linguist… find
  34. Qinglin Zhu, Bin Liang, Jingyi Sun, Jiachen Du, Lanjun Zhou, and Ruifeng Xu. 2022. Enhancing zero-shot stance detection via targeted background knowledge. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 2070–2075. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/akash-2025-can-large-language-models-address/