Elamur Research⌕
☰
home / papers / 2024 / CUFE at StanceEval2024: Arabic Stance Detection with Fine-Tuned Llama-3 Model

CUFE at StanceEval2024: Arabic Stance Detection with Fine-Tuned Llama-3 Model

Summary

The paper describes a system for the StanceEval 2024 shared task. Given an Arabic tweet and a target topic, the system decides whether the author is in favor, against, or neutral. The author takes Meta's 8B-parameter Llama-3 and adapts it with parameter-efficient fine-tuning. The base weights are loaded in 4-bit precision, LoRA adapters are trained on every linear projection, and each training item is an instruction-style prompt that names the target, gives the tweet, and asks for one of three stance labels.

Training data come from the Mawqif corpus of Arabic tweets on three Middle Eastern topics. The whole run, including test-set generation, took about an hour on one Colab A100. The official test result was a macro F1 of 0.7647. Performance was clearly weaker on the digital transformation target.

The work matters because a modest, cheaply adapted open LLM outperforms the Mawqif results reported for earlier feature-engineering and multi-task systems. However, the paper gives no in-house ablation or baseline comparison.

Key points

  • Model and setup:
    • Llama-3 8B, loaded with 4-bit quantization and trained with mixed float16/float32.
    • LoRA with rank 64, scaling factor 16 and dropout 0.05, applied to q/k/v/o and gate/up/down projections.
    • Adam optimizer, learning rate 1e-4, batch size 1, evaluation every 50 steps.
  • Data: Mawqif contains 4,121 tweets split across three targets: COVID-19 vaccine (1,373), digital transformation (1,348) and women empowerment (1,400). Each tweet carries stance, sentiment and sarcasm labels.
  • Mawqif annotation: each tweet–target pair received 3–7 annotations. Labeling stopped once the confidence score passed 0.7 or seven annotations had been collected.
  • Test results:
    • Macro F1 0.7647 overall.
    • Per target: women empowerment 0.8363, COVID-19 vaccine 0.8038, digital transformation 0.6539.
  • Earlier Mawqif results cited in the paper:
    • SMTL-HW, a multi-task model with sentiment and sarcasm as auxiliary tasks: macro F1 0.6865.
    • A GA feature selection plus ensemble approach: 68.76% F1 for stance.

Interesting ideas

  • I1. The prompt is conditioned on the target: it names the topic and restricts the reply to a fixed set of three label words, which makes one generative model behave like a constrained classifier that works across many topics. — For Mova (learn): LLM teacher tiers could be asked for silver labels using the same pattern, with the frame or relation stated in the prompt and only a closed label vocabulary allowed, so answers are easy to parse and to compare with witness parsers.
  • I2. Using a 4-bit base model with LoRA on all linear layers made it possible to adapt an 8B model within roughly an hour on a single GPU, which is cheap enough to repeat often. — For Mova (learn): a small local teacher could be specialized cheaply for one narrow labeling job, such as speech-act or stance tags, instead of relying only on general prompting.
  • I3. Reporting scores per target exposed a weak topic (digital transformation) that the overall average conceals. — For Mova (answer): test results for the readers could be broken down by text genre or question type, so a weak slice becomes visible rather than being masked by the mean.
  • I4. In the Mawqif annotation, each item gets more labels only until a confidence threshold or a maximum count is reached, which directs annotation effort to the hard cases. — For Mova (learn): when building silver data, more witness parsers or teacher calls could be requested only for sentences where the first witnesses disagree, stopping once agreement passes a threshold.
  • I5. The related work points out that sarcasm makes surface sentiment misleading about stance: many seemingly positive COVID tweets were in fact sarcastic. Treating sarcasm as an auxiliary signal improved stance detection in that work. — For Mova (frames): the irony module's output could be fed into evaluative frames, flipping or flagging the polarity of a speaker's attitude before it is passed on to the world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.93

@inproceedings{ibrahim2024cufe,
  title = {{CUFE at StanceEval2024: Arabic Stance Detection with Fine-Tuned Llama-3 Model}},
  author = {Ibrahim, Michael},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.93},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.93}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023. Enhancing stance detection through sequential weighted multi-task learning. Social Network Analysis and Mining, 14(1):7. find
  2. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024: The first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find our card
  3. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. find
  4. Samah M Alzanin, Abdu Gumaei, Md Azimul Haque, and Abdullah Y Muaad. 2023. An optimized arabic multilabel text classification approach using genetic algorithm and ensemble learning. Applied Sciences, 13(18):10264. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei https://github.com/MichaelIbrahim-GaTech/StanceEval2024.git Target Covid-19 Vaccine Tweet Stance Sentiment Positive Sarcasm No Favor Negative No None Neutral Yes  @YK. @ A J ®‚m' Bð Õæ ª¢ h. AJm ' AÓð YÒmÌ '@ é<Ëð Aî DÓ A JJ.£ð A KðPñ» A J   ƒAg Against we were diagnosed with Corona and recovered from it, thank God, we do not need a vaccination and we will never regret it Digital Trans. ø ñ Jƒ PYë éJ  Ò» ?i.ëA JÒÊË ú Gð QºËB@ ÈñjJË@ áK @ !!H. AJ» àñJ ÊÓ  I.JºË@ áÓ BYK. éJ  kñÊË@ è Qêk. B@ ÈCg@ ú æÒJ K é  ®ƒ ñÓ I.JºÊË Millon books!! Where is the digital transformation of curricula? The amount of annual waste of books is unfortunate. We wish to replace books with tablets Women Empow.   ¯ èñJ JË@ _ éJ «YÓ _ ú Ϋ _ ‘J ®Ë@# ¡Ê « è @QÖÏ @ á ºÖ ß  éÒëA . . \#Arrest_of_the_imposter_of_prophecy she misunderstood women’s empowerment *Table 1: Examples from the MAWQIF dataset* Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. ↗ our card
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  6. Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine-tuning design spaces. arXiv preprint arXiv:2301.01821. ↗
  7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  8. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. ↗
  9. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  10. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32. find our card
  11. Fatima Haouari and Tamer Elsayed. 2024. Are authorities denying or supporting? detecting stance of authorities towards rumors in twitter. Social Network Analysis and Mining, 14(1):34. find
  12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. find
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ibrahim-2024-cufe-stanceeval2024-arabic-stance-detection/