Elamur Research⌕
☰
home / papers / 2023 / Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification

Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification

Paper · 2023 · Iain J. Cruickshank and Lynnette Hui Xian Ng

Summary

Stance classification identifies the position an author takes toward a given target. Most existing systems depend on costly hand-labelled training data and transfer poorly to new topics. Cruickshank and Ng test whether open-source large language models can do this job with little or no annotation. Their benchmark crosses 10 openly available LLMs with 7 prompting strategies, including few-shot prompting (FSP) and chain-of-thought (CoT). They evaluate these combinations on 6 Twitter datasets that already carry human stance labels. They also fine-tune the models to see whether task specialization helps.

The results are mixed. LLMs can match in-domain supervised classifiers and sometimes beat them, but only with particular prompts. Their scores vary widely across models, prompts and datasets. Fine-tuning brought no reliable gain and sometimes reduced out-of-domain accuracy. The authors argue that LLM developers should treat stance detection as a standard benchmark. For anyone labelling opinion in social media, the study cautions against using LLMs as a simple replacement for supervised models.

Key points

  • Scale of evaluation: 10 open-source LLMs × 7 prompting schemes × 6 manually annotated Twitter stance datasets. Decoding was greedy, and FSP examples were chosen by hand.
  • LLMs are competitive with, but not consistently better than, smaller supervised baselines. They win only under specific prompt designs, so prompt wording matters a lot.
  • Among outputs where the model produced a valid stance label, roughly half (50.8%) were correct.
  • Fine-tuning did not guarantee improvement and could hurt out-of-domain performance. The authors therefore favor general models over specialized ones.
  • Practical advice: encoder-decoder architectures combined with FSP or CoT prompts gave the best results. The code is published at github.com/ijcruic/LLM-Stance-Labeling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️United States ArmyArlington · United States2 works · 291 citations

Also on the byline: Army Cyber Institute at West Point.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Iain J. Cruickshank and Lynnette Hui Xian Ng. 2023. Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification. arXiv:2309.13734.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.13734 · arXiv 2309.13734

@misc{cruickshank2023prompting,
  title = {{Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification}},
  author = {Iain J. Cruickshank and Lynnette Hui Xian Ng},
  year = {2023},
  doi = {10.48550/arxiv.2309.13734},
  eprint = {2309.13734},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.13734}
}

References

67 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023. Can we trust the evaluation on ChatGPT? arXiv preprint arXiv:2303.12767 (2023).
  2. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management 58, 4 (2021), 102597. find
  3. Abeer Aldayel and Walid Magdy. 2022. Characterizing the role of bots’ in polarized stance on social media. Social Network Analysis and Mining 12, 1 (2022), 30. find
  4. Rabab Alkhalifa and Arkaitz Zubiaga. 2022. Capturing stance dynamics in social media: open challenges and research directions. International Journal of Digital Humanities 3, 1-3 (2022), 115–135. find
  5. Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. arXiv preprint arXiv:2010.03640 (2020).
  6. Emily Allaway and Kathleen McKeown. 2023. Zero-shot stance detection: Paradigms and challenges. Frontiers in Artificial Intelligence 5 (2023), 1070429. find
  7. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023. A systematic review of machine learning techniques for stance detection and its applications. Neural Computing and Applications 35, 7 (2023), 5113–5144. find
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  9. Jiuhai Chen, Lichang Chen, Heng Huang, and Tianyi Zhou. 2023. When do you need Chain-of-Thought Prompting for ChatGPT? arXiv preprint arXiv:2304.03262 (2023). — [arXiv:2304.03262](https://arxiv.org/abs/2304.03262) find
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). find
  11. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-They-Won’t-They: A Very Large Dataset for Stance Detection on Twitter. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 1715–1724.
  12. Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020. Unsupervised user stance detection on Twitter. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 141–152. Manuscript submitted to ACM find
  13. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220–235.
  14. Heba Elfardy and Mona Diab. 2016. Cu-gwu perspective at semeval-2016 task 6: Ideological stance detection in informal text. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). 434–439. find
  15. Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023. Reasoning Implicit Sentiment with Chain-of-Thought Prompting. arXiv preprint arXiv:2305.11255 (2023). find
  16. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks Are All You Need. arXiv preprint arXiv:2306.11644 (2023). find
  17. Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 Misinformation on Social Media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020.
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  19. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). find
  20. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge Enhanced Masked Language Model for Stance Detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4725–4735.
  21. Kiana Kheiri and Hamid Karimi. 2023. SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning. arXiv preprint arXiv:2307.10234 (2023).
  22. Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task Learning for Rumour Verification. In Proceedings of the 27th International Conference on Computational Linguistics. 3402–3413. find
  23. Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 (2022). find
  24. Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2018. Stance evolution and twitter interactions in an italian political debate. In International Conference on Applications of Natural Language to Information Systems. Springer, 15–27. find
  25. Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. 2023. Stance detection with collaborative role-infused llm-based agents. arXiv preprint arXiv:2310.10467 (2023). find
  26. Ang Li, Bin Liang, Jingqian Zhao, Bowen Zhang, Min Yang, and Ruifeng Xu. 2023. Stance Detection on Social Media with Background Knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 15703–15717.
  27. Chandreen Liyanage, Ravi Gokani, and Vijay Mago. 2023. GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification Task. Authorea Preprints (2023). find
  28. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). find
  29. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 (2021).
  30. Mark Mets, Andres Karjus, Indrek Ibrus, and Maximilian Schich. 2023. Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media. arXiv preprint arXiv:2305.13047 (2023).
  31. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017). find
  32. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016). 31–41.
  33. Akiko Murakami and Rudy Raymond. 2010. Support or oppose? Classifying positions in online debates from reply activities and opinion expressions. In Coling 2010: Posters. 869–875. find
  34. Lynnette Hui Xian Ng and Kathleen M Carley. 2022. Is my stance the same as your stance? A cross validation study of stance detection datasets. Information Processing & Management 59, 6 (2022), 103070. find
  35. Lynnette Hui Xian Ng and Kathleen M Carley. 2022. Pro or Anti? a social influence model of online stance flipping. IEEE Transactions on Network Science and Engineering 10, 1 (2022), 3–19. find
  36. OpenAI. 2023. GPT-4 Technical Report. https://cdn.openai.com/papers/gpt-4.pdf. [Accessed 27-02-2024]. ↗
  37. Ron Korenblum Pick, Vladyslav Kozhukhov, Dan Vilenchik, and Oren Tsur. 2022. STEM: unsupervised structural embedding for stance detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 11174–11182. find
  38. Sunil Ramlochan. 2023. What is Prompt Engineering? https://www.promptengineering.org/what-is-prompt-engineering/. Accessed: 2023-09-14. ↗
  39. Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data programming: Creating large training sets, quickly. Advances in neural information processing systems 29 (2016). find
  40. Kevin Schaul, Szu Yu Chen, and Nitasha Tiku. 2023. Inside the Secret List of Websites that make AI like ChatGPT Sound Smart. https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ ↗
  41. Douglas C Schmidt, Jesse Spencer-Smith, Quchen Fu, and Jules White. [n. d.]. Cataloging Prompt Patterns to Enhance the Discipline of Prompt Engineering. ([n. d.]). find
  42. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150 (2019). Manuscript submitted to ACM find
  43. Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. 551–557.
  44. Marek Šuppa, Daniel Skala, Daniela Jašš, Samuel Sučík, Andrej Švec, and Peter Hraška. 2024. Bryndza at ClimateActivism 2024: Stance, Target and Hate Event Detection via Retrieval-Augmented GPT-4 and LLaMA. arXiv preprint arXiv:2402.06549 (2024). find
  45. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131 (2022).
  46. Surendrabikram Thapa, Kritesh Rauniyar, Farhan Ahmad Jafri, Shuvam Shiwakoti, Hariram Veeramani, Raghav Jain, Guneet Singh Kohli, Ali Hürriyetoğlu, and Usman Naseem. 2024. Stance and Hate Event Detection in Tweets Related to Climate Activism - Shared Task at CASE 2024. In Proceedings of the 7th Work… find
  47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  48. Apoorva Upadhyaya, Marco Fisichella, and Wolfgang Nejdl. 2023. A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 17. 854–865. find
  49. Ramon Villa-Cox, Sumeet Kumar, Matthew Babcock, and Kathleen M Carley. 2020. Stance in replies and quotes (srq): A new dataset for learning stance in twitter conversations. arXiv preprint arXiv:2006.00691 (2020). — [arXiv:2006.00691](https://arxiv.org/abs/2006.00691) find
  50. Ingmar Weber, Venkata R. Kiran Garimella, and Alaa Batayneh. 2013. Secular vs. Islamist Polarization in Egypt on Twitter. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (Niagara, Ontario, Canada) (ASONAM ’13). Association for Computing… — [doi:10.1145/2492517.2492557](https://doi.org/10.1145/2492517.2492557) find
  51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  52. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 (2023).
  53. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at semeval-2016 task 6: A specific convolutional neural network system for effective stance detection. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016). 384–388.
  54. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023).
  55. Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, et al. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12. find
  56. Ruoyao Yang, Wanying Xie, Chunhua Liu, and Dong Yu. 2019. BLCU_NLP at SemEval-2019 Task 7: An Inference Chain-based GPT Model for Rumour Evaluation. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 1090–… find
  57. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023). find
  58. Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023. Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240 (2023). — [arXiv:2308.01240](https://arxiv.org/abs/2308.01240) find
  59. Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). 458–463.
  60. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgpt? arXiv preprint arXiv:2212.14548 (2022). find
  61. Bowen Zhang, Daijun Ding, Liwen Jing, and Hu Huang. 2023. A Logically Consistent Chain-of-Thought Approach for Stance Detection. arXiv preprint arXiv:2312.16054 (2023). — [arXiv:2312.16054](https://arxiv.org/abs/2312.16054) find
  62. Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023. Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media. arXiv preprint arXiv:2304.03087 (2023). find
  63. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2023. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. In The Twelfth International Conference on Learning Representations. find
  64. Chong Zhang, Zhenkun Zhou, Xingyu Peng, and Ke Xu. 2023. DoubleH: Twitter User Stance Detection via Bipartite Graph Neural Networks. arXiv preprint arXiv:2301.08774 (2023). — [arXiv:2301.08774](https://arxiv.org/abs/2301.08774) find
  65. Guangzhen Zhao and Peng Yang. 2020. Pretrained embeddings for stance detection with hierarchical capsule network on social media. ACM Transactions on Information Systems (TOIS) 39, 1 (2020), 1–32. find
  66. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning. PMLR, 12697–12706.
  67. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] Manuscript submitted to ACM

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cruickshank-2023-prompting-fine-tuning-open-sourced/