Elamur Research⌕
☰
home / papers / 2024 / MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection

MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection

Summary

This StanceEval2024 shared-task entry tackles target-specific stance detection in Arabic tweets, an area the authors describe as under-researched despite its relevance to public opinion. They fine-tuned three instruction-tuned LLMs (GPT-3.5-Turbo-0125, Meta-Llama-3-8B-Instruct and Falcon-7B-Instruct) on the multi-dialectal MAWQIF corpus. Each tweet in MAWQIF is labelled Favor, Against or None toward one of three targets. Tweets were first cleaned with the AraBERT pre-processor. GPT-3.5 was tuned through OpenAI's hosted service. The two open models were adapted locally with LoRA via LitGPT on a single A100 GPU. Performance was measured as macro-F1 averaged over the Favor and Against classes. On the validation split, GPT-3.5-Turbo clearly led with 82.93 macro-F1. Llama-3 reached a mid-range score and was weak on Against tweets. Falcon nearly failed to recognize Against stances at all. The submission placed second on the blind-test leaderboard. The work suggests that lightweight fine-tuning can make general LLMs competitive for Arabic social-media opinion mining. It also shows that results differ widely across model families.

Key points

  • Data: MAWQIF comprises 4,121 Arabic tweets spanning multiple dialects, each addressing one of three subjects: Women Empowerment, the COVID-19 Vaccine or Digital Transformation. It also carries sentiment and sarcasm labels. The authors used a stance-stratified 85/15 train/validation split.
  • Training setup: All models were trained for 3 epochs. The open models used LoRA with rank 32, α 16, dropout 0.05, batch size 8, 10 warm-up steps and learning rate 2×10⁻⁴. Training took roughly 15–30 minutes per model.
  • Metric: Macro-F1 is the mean of the Favor and Against F1 scores. The None class is excluded because it is sparsely represented.
  • Validation results (macro-F1):
    • GPT-3.5-Turbo-0125 scored 82.93 overall, with 90.42 on Women Empowerment, 90.12 on COVID-19 Vaccine and 82.86 on Digital Transformation.
    • Meta-Llama-3-8B-Instruct scored 74.33 overall. Its Against F1 on Digital Transformation was only about 0.35.
    • Falcon-7B-Instruct scored 40.73 overall, with Against F1 of 3.92, 15.69 and 0.
  • Ranking and future work: The system ranked 2nd in StanceEval2024 on the blind test set. Proposed next steps include hyperparameter and instruction tuning and retrieval-augmented few-shot context (RAG).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.95

@inproceedings{alghaslan2024mgkm,
  title = {{MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection}},
  author = {Alghaslan, Mamoun and Almutairy, Khaled},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.95},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.95}
}

References

45 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. Lightning AI. 2023. Litgpt. https://github.com/ Lightning-AI/litgpt. ↗
  3. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023. Can we trust the evaluation on chatgpt? arXiv preprint arXiv:2303.12767. ↗
  4. Emily Allaway and Kathleen McKeown. 2023. Zero-shot stance detection: Paradigms and challenges. Frontiers in Artificial Intelligence, 5:1070429. ↗
  5. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023. A systematic review of machine learning techniques for stance detection and its applications. Neural Computing and Applications, 35(7):5113–5144. ↗
  6. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024: The first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find our card
  7. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. find
  8. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 1–9. find
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical stance detection for twitter: A two-phase lstm model using attention. In Advances in Information Retrieval: 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Proceedings 40, pages 529–536. Springer. find
  12. Heba Elfardy and Mona Diab. 2016. Cu-gwu perspective at semeval-2016 task 6: Ideological stance detection in informal text. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 434–439. find
  13. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. ↗
  14. Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In Social, Cultural, and Behavioral Modeling: 11th International Conference, SBP-BRiMS 2018, Washington, DC, USA, July 10-13, 2018, Proceedings 11, pages 197–206. Springer. find
  15. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 4725–4735. ↗ our card
  16. Kiana Kheiri and Hamid Karimi. 2023. Sentimentgpt: Exploiting gpt for advanced sentiment analysis and its departure from current machine learning. arXiv preprint arXiv:2307.10234. ↗
  17. Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2018. Stance evolution and twitter interactions in an italian political debate. In Natural Language Processing and Information Systems: 23rd International Conference on Applications of Natural Language to Information Systems, NLDB 2018, Paris, France, June 13-15, 2018, Proceedings 23, pages 15–27. Springer. find
  18. Douglas C Schmidt, Jesse Spencer-Smith, Quchen Fu, and Jules White. 2023. Cataloging prompt patterns to enhance the discipline of prompt engineering. URL: https://www.dre. vanderbilt. edu/˜ schmidt/PDF/ADA_Europe_Position_Paper. pdf [accessed 2023-09-25]. ↗
  19. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  20. Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. 2023. Stance detection with collaborative role-infused llm-based agents. arXiv preprint arXiv:2310.10467. ↗
  21. Marilyn A Walker, Pranav Anand, Rob Abbott, Jean E Fox Tree, Craig Martell, and Joseph King. 2012. That is your evidence?: Classifying stance in online political debate. Decision Support Systems, 53(4):719–729. ↗
  22. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayara-man Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365. find our card
  23. Ingmar Weber, Venkata R Kiran Garimella, and Alaa Batayneh. 2013. Secular vs. islamist polarization in egypt on twitter. In Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining, pages 290–297. find
  24. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. ↗ our card
  25. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846. ↗ our card
  26. Mark Mets, Andres Karjus, Indrek Ibrus, and Maximilian Schich. 2024. Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media. Plos one, 19(4):e0302380. ↗
  27. Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at semeval-2016 task 6: A specific convolutional neural network system for effective stance detection. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 384–388. ↗ our card
  28. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41. find
  29. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382. ↗ our card
  30. Lynnette Hui Xian Ng and Kathleen M Carley. 2022. Is my stance the same as your stance? a cross validation study of stance detection datasets. Information Processing & Management, 59(6):103070. ↗
  31. Ruoyao Yang, Wanying Xie, Chunhua Liu, and Dong Yu. 2019. Blcu_nlp at semeval-2019 task 7: An inference chain-based gpt model for rumour evaluation. In Proceedings of the 13th international workshop on semantic evaluation, pages 1090–1096. find
  32. Ron Korenblum Pick, Vladyslav Kozhukhov, Dan Vi-lenchik, and Oren Tsur. 2022. Stem: Unsupervised structural embedding for stance detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11174–11182. ↗
  33. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? Preprint, arXiv:2302.06476. ↗ our card
  34. Sunil Ramlochan. 2023. What is prompt engineering? https://www.promptengineering.org/ what-is-prompt-engineering/. ↗
  35. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? Preprint, arXiv:2305.18153. ↗
  36. Guido Zarrella and Amy Marsh. 2016. Mitre at semeval-2016 task 6: Transfer learning for stance detection. arXiv preprint arXiv:1606.03784. ↗ our card
  37. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgpt? arXiv preprint arXiv:2212.14548. ↗
  38. Bowen Zhang, Daijun Ding, Liwen Jing, and Hu Huang. 2023a. A logically consistent chain-of-thought approach for stance detection. arXiv preprint arXiv:2312.16054. ↗
  39. Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023b. Investigating chain-of-thought with chatgpt for stance detection on social media. arXiv preprint arXiv:2304.03087. ↗
  40. Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. arXiv preprint arXiv:1909.03477. ↗
  41. Chong Zhang, Zhenkun Zhou, Xingyu Peng, and Ke Xu. 2023c. Doubleh: Twitter user stance detection via bipartite graph neural networks. arXiv preprint arXiv:2301.08774. ↗
  42. Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao-jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hong-sheng Li, and Yu Qiao. 2023d. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. Preprint, arXiv:2303.16199. ↗ our card
  43. Guangzhen Zhao and Peng Yang. 2020. Pretrained embeddings for stance detection with hierarchical capsule network on social media. ACM Transactions on Information Systems (TOIS), 39(1):1–32. ↗
  44. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. Preprint, arXiv:2303.18223. ↗ our card
  45. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/alghaslan-2024-mgkm-stanceeval2024-fine-tuning-large/