Elamur Research⌕
☰
home / papers / 2024 / TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT

TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT

Summary

The paper describes TAO, a team entry for the StanceEval2024 shared task at ArabicNLP 2024. The task is to detect a tweet's stance toward a target (Favor, Against, None) in noisy, multi-dialectal Arabic social media text. The data is the Mawqif corpus: just over 4,000 tweets on three targets ("COVID-19 vaccine", "digital transformation", "women empowerment"). Each tweet is also annotated for sentiment and sarcasm, but TAO ignores those labels and uses a single-task setup.

After cleaning the tweets, the authors fine-tuned AraBERT v1 with a softmax classifier on the [CLS] vector. They chose v1 over v2 because it scored better in their own trials.

On their own 10% held-out split, TAO scored above 90 macro-F1 on every target. On the organizers' blind test set it dropped to between 64.55 and 73.30. The paper is inconsistent about which of the two lower scores belongs to COVID-19 versus digital transformation.

The work is a simple, reproducible baseline (the code is public). Its main lesson is how far an in-house evaluation can overestimate real performance.

Key points

  • Data. Mawqif has 4,121 tweets: 3,502 in train and 619 in test. Class balance differs by target:

    • COVID-19 vaccine is roughly balanced between Favor and Against (about 43.6% each).
    • Digital transformation (about 77% Favor) and women empowerment (about 64% Favor) lean heavily toward Favor.
    • The None class is rare: about 5–13% depending on target.

  • Preprocessing. The authors removed:

    • numbers, Latin letters and newlines;
    • any token containing "_", "#" or "@";
    • diacritics, Tatweel and other non-Arabic characters.

  • Model and training. AraBERT v1 (BERT-Base), using the BertTokenizer and Hugging Face TrainingArguments. Settings: batch size 16 for training and 32 for evaluation, 3 epochs, max length 128, learning rate in the 2e-5 to 5e-5 range. Training ran on Google Colab.

  • Internal results. On a 70/20/10 train/validation/test split, macro-F1 was:

    • COVID-19 vaccine: 90.36;
    • the digital transformation target: 92.36;
    • the women empowerment target: 91.34.

    These beat CAMeLBERT-da, MARBERT, AraBERT and AraBERT-twitter as reported by Alturayeif et al. (2022). AraBERT v2 reached 73.05, 60.14 and 82.02 on the same targets.

  • Blind-test results. Scores were 73.30 for women empowerment, with 70.51 and 64.55 for the other two targets (the assignment is swapped between abstract and results). The official macro-F1 averages only the Favor and Against F1 scores.

Interesting ideas

  • I1. The official metric averages F1 over the two opinion-bearing classes and leaves out the rare None class, so scoring focuses on the decisions that matter rather than on the neutral class. — For Mova (answer): Test scoring could report accuracy on content-bearing answers separately from "no answer / unknown" cases, so abstentions neither inflate nor hide quality.
  • I2. A score above 90 on a self-made split fell to 64–73 on the organizers' blind set, which shows how misleading a homemade split can be. — For Mova (learn): Always keep a held-out set built by a different process (other source, other annotator) as a control, and treat large gaps between internal and external scores as a warning of leakage or overfitting.
  • I3. The authors checked that per-target class proportions in train and test closely match before training, a cheap sanity check on the split. — For Mova (learn): When building silver data or splits from treebanks, automatically compare label distributions (deprels, frame types) across splits and flag skewed ones.
  • I4. Social-media noise was handled by aggressively stripping hashtags, mentions, diacritics and foreign script before tokenization. This trades information for cleaner input. — For Mova (tokens): A configurable normalization layer for noisy English input (handles, hashtags, emoji) could sit before the tokenizer, but it should record what was removed so later stages stay explainable.
  • I5. The corpus labels stance, sentiment and sarcasm on the same tweets, which opens the way to studying how these attitudes interact (the paper defers multi-task use to future work). — For Mova (frames): Speech-act and irony modules in prag could be evaluated jointly with stance-like attitude labels, checking whether detected irony correctly flips the inferred attitude of the speaker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.100

@inproceedings{melhem2024tao,
  title = {{TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT}},
  author = {Melhem, Anas and Hamed, Osama and Sammar, Thaer},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.100},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.100}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training BERT on arabic tweets: Practical considerations. CoRR, abs/2102.10684. ↗ our card
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  3. Tahani Alqurashi. 2022. Stance analysis of distance education in the kingdom of saudi arabia during the COVID-19 pandemic using arabic twitter data. Sensors, 22(3):1006. find
  4. Reema Khaled AlRowais and Duaa AlSaeed. 2023. Arabic stance detection of covid-19 vaccination using transformer-based approaches: a comparison study. Arab Gulf Journal of Scientific Research. find
  5. Nora Saleh Alturayeif, Hamzah Luqman, and Moataz A. Ahmed. 2023. Enhancing stance detection through sequential weighted multi-task learning. Soc. Netw. Anal. Min., 14(1):7. find
  6. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  7. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831, Online. Association for Computational Linguistics. find
  8. Imane Guellil, Houda Saadane, Faiçal Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. J. King Saud Univ. Comput. Inf. Sci., 33(5):497–507. find our card
  9. Fatima Haouari and Tamer Elsayed. 2023. Detecting stance of authorities towards rumors in arabic tweets: A preliminary study. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part II, volume 13981 of Lecture Notes in Computer Science, pages 430–438. Springer. find our card
  10. Abdullah Al Imran and Md Nur Amin. 2021. Deep bangla authorship attribution using transformer models. In Computational Data and Social Networks -10th International Conference, CSoNet 2021, Virtual Event, November 15-17, 2021, Proceedings, volume 13116 of Lecture Notes in Computer Science, pages 118–128. Springer. find
  11. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2024. A survey on arabic named entity recognition: Past, recent advances, and future trends. IEEE Trans. Knowl. Data Eng., 36(3):943–959. find our card
  12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  14. Gilbert Badaro, Ramy Baly, Hazem M. Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al Sallab, and Ali Hamdi. 2019. A survey find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/melhem-2024-tao-stanceeval2024-shared-task-arabic/