Elamur Research⌕
☰
home / papers / 2024 / SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection

SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection

Summary

This paper reports Team SMASH's entry to the StanceEval 2024 shared task. The task asks whether the author of an Arabic text is for, against, or neutral toward a given target. The data is Mawqif, an Arabic corpus of 4,121 sentences on three topics, annotated with stance, sentiment and sarcasm. The authors compare two approaches:

  • Fine-tuning Arabic and multilingual encoders: AraBERT, MARBERT, CAMeLBERT, QARiB, mBERT and AraBART.
  • Zero-shot prompting of quantized LLMs run through Ollama: Llama 3, Command-R, Command-R+, Gemma, Mistral, Mixtral, AceGPT and WizardLM-2.

The prompt design that worked best gives instructions in English, names the target in Arabic, and requires a fixed English label. Asking for Arabic output led to rambling, non-committal answers. MARBERT, which was pretrained on tweets, was the strongest fine-tuned model. Command-R in zero-shot mode beat every model and placed 4th officially. The result shows that a large multilingual LLM, prompted carefully, can outperform task-specific Arabic encoders on a low-resource stance task.

Key points

  • Data split: Mawqif covers COVID-19 vaccine, Digital Transformation and Women Empowerment. The authors' split was 2,801 train, 701 validation and 619 test sentences.
  • Metric: the official score is the mean of the F1 scores for the Favor and Against classes. The Neutral class is excluded.
  • Fine-tuning setup: 7 epochs, AdamW, learning rate 2e-05, batch size 128, weight decay 0.01.
  • Validation results: MARBERT reached F1 0.570 and macro F1 0.770. Command-R (35b-v0.1-q8_0) reached F1 0.661 and macro F1 0.820. The official submission scored 0.8041 and ranked 4th.
  • Upsampling experiment: oversampling the Against class for MARBERT slightly lowered results (0.543 / 0.757). Larger LLMs (Command-R, Llama 3 70B) clearly outperformed the 7–8B models.

Interesting ideas

  • I1. Use the LLM only to read the text, and force its answer into a closed English label set, because free-form generation in the weaker language produced hallucinated replies. — For Mova (learn): LLM teacher tiers could be queried with closed label inventories (e.g., role names, frame IDs) rather than free text, which makes their silver labels cheaper to check and agree on.
  • I2. Add a tolerant post-processing step that maps raw LLM output to labels, catching spelling variants (favour/favor) and answers that mix scripts. — For Mova (learn): a small, explicit normalizer for teacher outputs would reduce silently dropped or mislabeled silver examples, and its mapping rules stay inspectable.
  • I3. Treat stance as a judgment about a specific (text, target) pair, kept separate from sentiment and sarcasm, which Mawqif annotates as distinct labels on the same sentence. — For Mova (frames): prag could compute the author's attitude toward a named entity as its own frame slot, rather than reusing overall sentiment, which matters when irony flips the surface polarity.
  • I4. Compare per-class confusion matrices of the two best systems. Command-R was stronger on Against and Neutral, while MARBERT over-predicted Favor, so the two have complementary error profiles. — For Mova (learn): when choosing witness parsers or teachers for silver agreement, prefer ones whose per-label errors differ, and track per-label confusions instead of a single score.
  • I5. Report negative results honestly: the obvious fix for class imbalance (upsampling) did not help. — For Mova (learn): evaluation should keep such "plausible fix, no gain" checks as controls, so that repair rules or data tweaks are adopted only when they beat the baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Sentiment & emotion · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.92

@inproceedings{hariri2024smash,
  title = {{SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection}},
  author = {Hariri, Youssef Al and Farha, Ibrahim Abu},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.92},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.92}
}

References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. find our card
  2. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591, Portland, Oregon, USA. Association for Computational Linguistics. find
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  4. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. find our card
  5. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  6. Ibrahim Abu Farha and Walid Magdy. 2021. A comparative study of effective approaches for arabic sentiment analysis. Information Processing & Management, 58(2):102438. ↗
  7. Tariq Alhindi, Amal Alabdulkarim, Ali Alshehri, Muhammad Abdul-Mageed, and Preslav Nakov. 2021. AraStance: A multi-country and multi-domain dataset of Arabic stance detection for fact checking. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 57–65, Online. Association for Computational Linguistics. find our card
  8. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024:the first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find our card
  9. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  11. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J. Jansen. 2020. Improving Arabic text categorization using transformer training diversification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 226–236, Barcelona, Spain (Online). Association for Computational Linguistics. find
  12. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗ our card
  14. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  15. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic. Preprint, arXiv:2309.12053. ↗ our card
  16. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. find our card
  17. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  18. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31–42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  19. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1). find
  20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗ our card
  21. AI at Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-stanceeval-2024-prompt-engineering/