Elamur Research⌕
☰
home / papers / 2024 / Team_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection

Team_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection

Summary

The paper addresses stance detection on Arabic tweets, which means deciding whether a tweet supports, opposes, or is neutral toward a target. The work was a StanceEval2024 shared task entry. The authors avoid full fine-tuning and instead treat pre-trained language models as frozen feature extractors with a cheap classifier on top.

They used the Mawqif dataset, which has three targets: women empowerment, COVID vaccine, and digital transformation. Three families of features were compared:

  • nine multilingual sentence transformers;
  • five Arabic BERT variants, each pooled two ways ([CLS] vector or mean over all output tokens), with every embedding fed to scikit-learn logistic regression;
  • a multi-task Parallel-Sum (P-SUM) head on a frozen MARBERT, trained jointly on stance and sentiment.

Arabic, tweet-pretrained encoders with mean pooling clearly outperformed the multilingual sentence models. The final system was a majority vote of the three strongest configurations. It scored 78.62 F avg2 on the blind test set and placed fifth of 16 teams. This suggests competitive results are possible without fine-tuning's compute and time cost.

Key points

  • Data: Mawqif has 3502 training tweets and 619 test tweets, labelled FAVOR / AGAINST / NONE, plus sentiment and sarcasm annotations.
    • Overall training distribution is roughly 61% favor, 29% against, 10% none.
    • Digital transformation is especially skewed, with about 77% favor.
  • Metric (F avg2): for each target, average the F1 of FAVOR and AGAINST only, then average across the three targets.
  • Internal test (500 held-out examples):
    • The best sentence transformer was paraphrase-multilingual-mpnet-base-v2 at 70.6; all-mpnet-base-v2 was lowest at 51.8.
    • MARBERT with averaged embeddings reached 76.7, P-SUM-MTL 76.3, and AraBERTv0.2-Twitter AVG 76.1.
  • Pooling: mean pooling beat [CLS] for all five Arabic BERTs tested (AraBERTv0.2-Twitter, MARBERT, Qarib, ARBERT, Arabic-BERT). For example, MARBERT went from 71.5 with [CLS] to 76.7 with averaging.
  • Official test: the ensemble scored 85.99 on women empowerment, 73.08 on COVID vaccine, 76.8 on digital transformation, and 78.62 overall (5th of 16).
  • Overfitting: this was the main difficulty, caused by the small data split across three targets. One model on all targets beat per-target models for the sentence-transformer setup.

Interesting ideas

  • I1. Averaging every token output of a frozen encoder gave a consistently better sentence vector than the [CLS] token, at no extra cost; this is a cheap default worth checking wherever frozen embeddings are pooled. — For Mova (tree): when gparse or auxiliary classifiers pool frozen transformer states for span- or sentence-level decisions, compare mean pooling against first-token pooling before adding parameters.
  • I2. A frozen encoder plus logistic regression is a fast, honest baseline that reveals how much a heavier trained head actually adds (here P-SUM barely moved past mean pooling). — For Mova (learn): keep a frozen-features + linear-probe baseline in evaluation so any gain from a new trained component is measured against what the raw representation already gives.
  • I3. Genre match of the pretraining corpus mattered more than whether the model was trained specifically for sentence embeddings, since tweet-pretrained Arabic BERTs beat purpose-built multilingual sentence models. — For Mova (tokens): when choosing or training encoders for tagging and lemmatization of fables, school texts, or word problems, prefer models whose pretraining genre resembles those texts over generic "sentence" models.
  • I4. Pooling training data across targets worked better than per-target models, because the way people signal support or opposition carries over between topics. — For Mova (frames): train speech-act, stance, or irony detectors in prag jointly across topics rather than per domain, treating the linguistic cues as shared and topic-independent.
  • I5. A simple majority vote over the three best, architecturally different systems produced the final submission. — For Mova (learn): the same vote over diverse witness parsers can decide silver labels, keeping only arcs or frames where most witnesses agree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.89

@inproceedings{galal2024team,
  title = {{Team\_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection}},
  author = {Galal, Omar and Kaseb, Abdelrahman},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.89},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.89}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. find our card
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  3. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024: The first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find our card
  4. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  6. David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20(2):215–232. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Hugo Touvron et al. 2023. Llama 2: Open foundation and fine-tuned chat models. find
  9. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852. ↗ our card
  10. Omar Galal, Ahmed H Abdel-Gawad, and Mona Farouk. 2024a. Federated freeze bert for text classification. Journal of Big Data, 11(1):28. find
  11. Omar Galal, Ahmed H Abdel-Gawad, and Mona Farouk. 2024b. Rethinking of bert sentence embedding for text classification. find
  12. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. find our card
  13. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  14. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
  15. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. find our card
  16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. ↗ our card
  17. Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2020. Improving bert performance for aspect-based sentiment analysis. arXiv preprint arXiv:2010.11731. ↗
  18. Abdelrahman Kaseb and Mona Farouk. 2022. SAIDS: A novel approach for sentiment analysis informed of dialect and sarcasm. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 22–30, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  19. Abdelrahman Kaseb and Mona Farouk. 2023. Active learning for arabic sentiment analysis. Alexandria Engineering Journal, 77:177–187. find
  20. Abdelrahman Kaseb, Omar Galal, and Dina Elreedy. 2022. Analysis on tweets towards covid-19 pandemic: An application of text-based depression detection. In 2022 4th Novel Intelligent and Leading Emerging Sciences Conference (NILES), pages 131– 136. find
  21. Hamdy Mubarak, Sabit Hassan, Shammur Absar Chowdhury, and Firoj Alam. 2022. Arcovidvac: Analyzing arabic tweets about covid-19 vaccination. arXiv preprint arXiv:2201.06496. ↗ our card
  22. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/galal-2024-team-zero-stanceeval2024-frozen-plms/