Elamur Research⌕
☰
home / papers / 2023 / AAST-NLP at ArAIEval Shared Task: Tackling Persuasion technique and Disinformation Detection using Pre-Trained Language Models On Imbalanced Datasets

AAST-NLP at ArAIEval Shared Task: Tackling Persuasion technique and Disinformation Detection using Pre-Trained Language Models On Imbalanced Datasets

Paper · 2023 · Ahmed El-Sayed, Omar Nasr and Noureldin Elmadany

Summary

The AAST-NLP team took part in the ArAIEval 2023 shared tasks on Arabic text. Task 1 is persuasion technique detection: 1A asks whether a tweet or paragraph is persuasive at all, and 1B tags it with any of 24 techniques. Task 2 is disinformation detection: 2A is binary, and 2B sorts disinformation into HS, OFF, Spam or Rumour.

All four datasets are heavily skewed. The team cleaned the text by removing stop words, tweet tags, diacritics, tatweel, emojis and letter-separating underscores. They then compared several Arabic BERT variants and chose AraBERT.

To cope with the skew, they tried three remedies:

  • random over- and under-sampling,
  • contextual-embedding augmentation via NLPAUG,
  • reweighted losses: weighted cross-entropy, focal loss and Dice loss.

Dice loss worked best for the single-label subtasks. The multi-label subtask instead used per-label binary cross-entropy. Final rankings were 11th (1A), 3rd (1B), 2nd (2A) and 3rd (2B). The paper is a compact, practical case study of which imbalance remedies do and do not help fine-tuned encoders on small, skewed classification data.

Key points

  • Data sizes:
    • Task 1 has 2427 train, 259 dev and 503 test texts.
    • Task 2A has 14,147, 2115 and 3729 tweets; Task 2B has 2648, 396 and 876.
    • In 2B, Rumour is rare and HS dominates.
  • Model choice: AraBERT beat MARBERT, ARBERT and bert-base-arabic on all four subtasks in development experiments.
  • Training setup: ktrain autofit with a triangular learning-rate schedule, learning rate 1e-5, batch size 16, max length 128, AdamW, early-stopping patience 5, and Dice smoothing 1e-6, on a Colab T4 GPU.
  • Test micro-F1: 0.7237 (1A), 0.5522 (1B), 0.9043 (2A) and 0.8253 (2B). The official random baselines were 0.4771, 0.0868, 0.5154 and 0.2603.
  • What failed:
    • Augmented data, even after manual filtering, sharply lowered micro-F1.
    • Focal loss led to overfitting.
    • A CNN-BiLSTM + AraBERT hybrid also overfit.

Interesting ideas

  • I1. Dice loss scores predictions by their overlap with the gold labels, so frequent easy negatives carry less weight; the authors found it more useful than class-weighted cross-entropy or focal loss on skewed labels.
  • I2. Masked-LM word substitution can silently flip a sentence's meaning, and here even manually filtered synthetic data hurt the scores, which is a strong warning against unchecked augmentation.
  • I3. The multi-label technique task was split into one independent binary classifier per label, while the authors noted that real dependencies between labels remain unmodelled.
  • I4. Reporting the official random baseline next to system scores shows how much is actually learned beyond chance, especially for the 24-label task, where chance sits near 0.09.
  • I5. Normalisation removed an odd formatting habit of underscores placed between every letter, which would otherwise wreck subword tokenization.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): rare classes such as irony, unusual speech acts or uncommon VerbNet roles could be trained with a Dice-style objective so the dominant "neutral/none" label does not swamp them.
  • I2 (learn): any synthetic or silver example should pass a meaning-preservation check, such as agreement of frames or graphs before and after the edit, before it enters training.
  • I3 (frames): implicature and persuasion-like cues could start as per-cue detectors, with a later explainable layer that captures how cues co-occur.
  • I4 (learn): every evaluation table could include a random or shuffled-label negative control as a fixed row.
  • I5 (tokens): the en tokenizer could detect and collapse letter-by-letter spacing or separators (e.g., "w_o_r_d") before tagging and lemmatisation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed El-Sayed, Omar Nasr and Noureldin Elmadany. 2023. AAST-NLP at ArAIEval Shared Task: Tackling Persuasion technique and Disinformation Detection using Pre-Trained Language Models On Imbalanced Datasets. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.56

@inproceedings{elsayed2023aast,
  title = {{AAST-NLP at ArAIEval Shared Task: Tackling Persuasion technique and Disinformation Detection using Pre-Trained Language Models On Imbalanced Datasets}},
  author = {Ahmed El-Sayed, Omar Nasr and Noureldin Elmadany},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.56},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.56}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022b. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Arabert: Transformer-based model for arabic language understanding. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
  5. Pascal Geldsetzer. 2020. Knowledge and Perceptions of COVID-19 among the general public in the United States and the United Kingdom: a cross-sectional online survey. Annals of Internal Medicine, 173(2):157– 160. Workshop, pages 357–363, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. Mohamed S. Kraiem, F. Sánchez, and María N. Moreno García. Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties. an approach based on association models. (18):8546. ↗
  6. Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2019. Dice Loss for Data-imbalanced NLP Tasks. arXiv (Cornell University).
  7. Arun S. Maiya. 2020. ktrain: A low-code library for augmented machine learning. arXiv preprint arXiv:2004.10703. ↗
  8. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. AbdelRahim A. Elmadany Muhammad Abdul-Mageed and El Moatez Billah Nagoudi. Arbert & marbert: Deep bidirectional transformers for arabic. ↗
  9. Ozgur Ozdemir and Elena Battini Sonmez. 2020. Weighted Cross-Entropy for Unbalanced Data with Application on COVID X-ray images. 2020 Innovations in Intelligent Systems and Applications Conference (ASYU). find
  10. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825– 2830. ↗
  11. Richard Perloff. 2017. The dynamics of persuasion: Communication and attitudes in the 21st century. find
  12. Irina Romanova and Irina Smirnova. 2019. Persuasive techniques in advertising. Training Language and Culture, 3:55–70. ↗
  13. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  14. Lauren Smith. 2015. Cyclical learning rates for training neural networks. arXiv (Cornell University). find
  15. Amey Hengle, Atharva Kshirsagar, Shaily Desai, and Manisha Marathe. 2021. Combining context-free and contextualized representations for Arabic sarcasm detection and sentiment identification. In Proceedings of the Sixth Arabic Natural Language Processing 569 5 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sayed-2023-aast-nlp-araieval-shared-task/