Elamur Research⌕
☰
home / papers / 2024 / StanceEval 2024: The First Arabic Stance Detection Shared Task

StanceEval 2024: The First Arabic Stance Detection Shared Task

Summary

StanceEval 2024 was the first shared task on stance detection for Arabic, held as part of the ArabicNLP 2024 conference. Stance detection asks whether a text's author supports, opposes, or takes no position on a target. It differs from sentiment analysis because the target may be abstract or never named, and a tweet's emotional tone can conflict with its stance. Participants built systems that assigned each tweet a Favor, Against, or None label with respect to one of three targets: women empowerment, the COVID-19 vaccine, or digital transformation. Data came from the Mawqif corpus, provided by the organizers, whose tweets also carry sentiment and sarcasm labels. Teams could therefore train on stance alone or use multi-task learning. Systems were ranked by macro F1 over the Favor and Against classes only. Most teams fine-tuned Arabic BERT variants such as MARBERT and AraBERT, while others used ensembles, multi-task setups, or prompted and fine-tuned LLMs. The best system reached an F-score of 84.38. The task gives Arabic NLP, where stance work has been scarce, a common benchmark and an overview of what currently works.

Key points

  • Participation: 28 teams registered, 16 submitted valid test runs, and 13 wrote system description papers.
  • Data: The Mawqif dataset contains 4,121 Arabic tweets, about 1.35–1.4k per topic. It was split 85/15 into 3,502 training and 619 blind test tweets, and the "Favor" label dominates, especially for digital transformation (about 77%).
  • Annotation quality: Each tweet was labeled by 3–7 annotators, accepted once confidence exceeded 0.7, and annotators who scored under 80% on test questions were excluded.
  • Evaluation: The metric averages the F1 scores of Favor and Against per target. The None class was used in training but not scored.
  • Systems: A fine-tuned GPT-3.5-Turbo-0125 placed second with 82.06 F1. MARBERT was the strongest backbone for several teams, including PICT and Rasid, and SMASH found that zero-shot LLMs did better with English prompts and labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.88

@inproceedings{alturayeif2024stanceeval,
  title = {{StanceEval 2024: The First Arabic Stance Detection Shared Task}},
  author = {Alturayeif, Nora and Luqman, Hamzah and Alyafeai, Zaid and Yamani, Asma},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.88},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.88}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗ our card
  2. Mamoun Alghaslan and Khaled Almutairy. 2024. Mgkm at stanceeval2024: Fine-tuning large language models for arabic stance detection. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find our card
  3. Rabab Alkhalifa, Elena Kochkina, and Arkaitz Zubiaga. 2021. Opinions are made to be changed: Temporally adaptive stance classification. In Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks, HT ’21. ACM. ↗
  4. Eman Sweiaad S. Alofi and Sami Mnasri. 2024. Stancealert: Effective logistic regression for stance detection. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  5. Nouf AlShenaifi, Nourah Alangari, and Hadeel Al-Negheimish. 2024. Rasid at stanceeval: Fine-tuning marbert for arabic stance detection. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  6. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023a. Enhancing stance detection through sequential weighted multi-task learning. ↗
  7. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023b. A systematic review of machine learning techniques for stance detection and its applications. Neural Computing and Applications, 35(7):5113–5144. ↗
  8. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. find
  9. Mezghani Amal, Rahma Boujelbane, and Mariem Ellouze. 2024. Anlp rg at stanceeval2024: Comparative evaluation of stance, sentiment and sarcasm detection. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv. ↗ our card
  11. Mohamed Badran, Mo’men Hamdy, Marwan Torki, and Nagwa El-Makky. 2024. Alexunlp-bh at stanceeval2024: Multiple contrastive losses ensemble strategy with multi-task learning for stance detection in arabic. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  12. Iain J. Cruickshank and Lynnette Hui Xian Ng. 2024. Prompting and fine-tuning open-sourced large language models for stance classification. Preprint, arXiv:2309.13734. ↗
  13. Omar Galal and Abdelrahman Kaseb. 2024. Team_zero at stanceeval2024: Frozen plms for arabic stance detection. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). ↗
  14. Manuela Gómez-Suta, Julián Echeverry-Correa, and José A. Soto-Mejía. 2023. Stance detection in tweets: A topic modeling approach supporting explainability. Expert Systems with Applications, 214:119046. ↗
  15. Anant Khandelwal. 2021. Fine-tune longformer for jointly predicting rumor stance and veracity. In Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD), CODS COMAD 2021. ACM. find
  16. Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3402–3413, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  17. Mirko Lai, Marcella Tambuscio, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2019. Stance polarity in political debates: A diachronic perspective of network homophily and conversations on twitter. Data & Knowledge Engineering, 124:101738. ↗
  18. Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. 2024. Stance detection with collaborative role-infused llm-based agents. Preprint, arXiv:2310.10467. ↗
  19. Mohamed Lichouri, Khaled Lounnas, OUARAS Khelil Rafik, Mohamed ABi, and Anis Guechtouli. 2024. dzstance@stanceeval2024: Arabic stance detection based on sentence transformers. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  20. Youssef Al Hariri and Ibrahim Abu Farha. 2024. Smash at stanceeval 2024: Prompt engineering llms for arabic stance detection. In Proceedings of the Arabic-NLP 2024, Bangkok, Thailand (Hybrid). find
  21. Yujian Liu, Xinliang Frederick Zhang, David Wegs-man, Nick Beauchamp, and Lu Wang. 2022. Politics: Pretraining with same-story article comparison for ideology prediction and stance detection. Preprint, arXiv:2205.00619. ↗ our card
  22. Ahmed Abul Hasanaath and Aisha Alansari. 2024. Stancecrafters at stanceeval2024: Multi-task stance detection using bert ensemble with attention based aggregation. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  23. Anas Melhem, Osama Hamed, and Thaer Sammar. 2024. Tao at stanceeval2024 shared task: Arabic stance detection using arabert. In Proceedings of the Arabic-NLP 2024, Bangkok, Thailand (Hybrid). find
  24. Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics. find
  25. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Linguistics. find
  26. Michael Ibrahim. 2024. Cufe at stanceeval2024: Arabic stance detection with fine-tuned llama-3 model. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). ↗
  27. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. Semeval-2016 task 6: Detecting stance in tweets. 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41. find
  28. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗ our card
  29. Mustapha Jaballah. 2024. Ishfmg_tun at stanceeval: Ensemble method for arabic stance evaluation system. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). ↗
  30. Mitra Mohtarami, James Glass, and Preslav Nakov. 2019. Contrastive language adaptation for cross-lingual stance detection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4442–4452, Hong Kong, China. Association for Computational Linguistics. ↗
  31. Ishaan Shukla, Ankit Vaidya, and Geetanjali Vinayak Kale. 2024. Pict at stanceeval2024: Stance detection in arabic using ensemble of large language models. In Proceedings of the ArabicNLP 2024, Bangkok, Thailand (Hybrid). find
  32. Qingying Sun, Xuefeng Xi, Jiajun Sun, Zhongqing Wang, and Huiyan Xu. 2022. Stance detection with a multi-target adversarial attention network. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(2):1–21. ↗
  33. Ingmar Weber, Venkata R. Kiran Garimella, and Alaa Batayneh. 2013. Secular vs. islamist polarization in egypt on twitter. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’13. ACM. find
  34. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  35. Yuanyu Yang, Bin Wu, Kai Zhao, and Wenying Guo. 2020. Tweet stance detection: A two-stage dc-bilstm model based on semantic attention. In 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC). IEEE. find
  36. Kai Ye, Yangheran Piao, Kun Zhao, and Xiaohui Cui. 2021. Graph Enhanced BERT for Stance-Aware Rumor Verification on Social Media, page 422–435. Springer International Publishing. find
  37. Chong Zhang, Zhenkun Zhou, Xingyu Peng, and Ke Xu. 2023. Doubleh: Twitter user stance detection via bipartite graph neural networks. Preprint, arXiv:2301.08774. ↗
  38. Shengping Zhou, Junjie Lin, Lianzhi Tan, and Xin Liu. 2019. Condensed convolution neural network by attention over self-attention for stance detection in twitter. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/