Elamur Research⌕
☰
home / papers / 2023 / DetectiveRedasers at ArAIEval Shared Task: Leveraging Transformer Ensembles for Arabic Deception Detection

DetectiveRedasers at ArAIEval Shared Task: Leveraging Transformer Ensembles for Arabic Deception Detection

Paper · 2023 · Bryan Tuck, Fatima Zahra Qachfar, Dainis Boumber et al.

Summary

The paper describes the system that ranked first in both subtasks of ArAIEval's second shared task, which focused on detecting deception in Arabic and was held at ArabicNLP 2023. Subtask 2A is a binary decision on whether an Arabic tweet is disinformative. Subtask 2B assigns disinformative tweets to one of four classes: hate speech, offensive, rumor or spam. Tweets are short, mix dialects, and often switch into English or French.

The authors first normalise the text. Foreign-language fragments are detected and machine-translated into Arabic, and emojis are replaced by Arabic textual descriptions. They then tune several Arabic BERT variants with Optuna and test a soft-voting ensemble of five identically configured models that differ only in random seed. The best single systems were AraBERT-Covid19 for 2A and AraBERTv02-Twitter for 2B.

On the test set, the single models slightly beat their ensembles in both subtasks. The authors conclude that ensembling is not automatically safer and needs its own validation. The 2B macro-F1 fell sharply from validation to test, which points to weak handling of the rare classes.

Key points

  • Data size and balance. Task 2A has 14,147 training, 2,115 dev and 3,729 test tweets, of which only about 19% are disinformative. Task 2B training data is small and skewed: 1,512 hate speech, 500 offensive, 453 spam and 191 rumor examples.
  • Candidate models. AraBERTv02-Twitter, AraBERT-Covid19, QARiB, MARBERTv2 and CAMeLBERT-DA SA.
  • Tuning setup. Optuna ran 100 trials using the TPE sampler and a Hyperband pruner, with each trial lasting 2–12 epochs and optimising micro-F1. The search space was:
    • learning rate 1e-5 to 5e-5;
    • batch size 8 to 64;
    • dropout 0 to 0.5;
    • maximum length 32 to 128.
  • Test results.
    • Task 2A, single AraBERT-Covid19: micro-F1 90.48%, macro-F1 86.26%. The ensemble reached 90.02% and 85.84%.
    • Task 2B, single AraBERTv02-Twitter: micro-F1 84.36%, macro-F1 75.51%. The ensemble reached 83.56% and 75.41%.
    • In 2B, macro-F1 dropped from about 81–82% on validation to about 75% on test.
  • Training details. AdamW with cross-entropy loss, mixed precision (AMP), gradient clipping at norm 1.0, early stopping, and no learning-rate scheduler. The ensemble was built with TorchEnsemble and its own hyperparameters were not tuned.

Interesting ideas

  • I1. Emojis as words. Instead of deleting emojis, each one is rewritten as a short descriptive phrase in the target language using CLDR annotation names, so the model can read its sentiment as ordinary text.
  • I2. Normalising foreign spans. Fragments in another language are detected per segment and normalised into the main language before modelling, so downstream components only face one language.
  • I3. Treating the ensemble as a separate system. A seed ensemble of identical models must be validated on its own, because in this paper it won on dev for 2B but lost on test in both subtasks.
  • I4. Watching macro-F1 for minority classes. The large validation-to-test gap in macro-F1, while micro-F1 stayed stable, exposed that the rare classes were being lost.
  • I5. Pruned tuning searches. Combining TPE sampling with Hyperband pruning lets a 100-trial search drop weak configurations after only a few epochs, which keeps hyperparameter search affordable for transformer fine-tuning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tokenizer could map emojis and similar symbols to CLDR English names tagged as special lexical items, so the tagger and later stages see a meaningful word rather than an unknown glyph.
  • I2 (tokens): A span-level language detector could mark foreign fragments in English input, so the morphology and lemma steps skip or specially handle them instead of producing garbage analyses.
  • I3 (learn): When several witness parsers vote to produce silver data, the agreement scheme itself (number of voters, weighting) should be tested on a held-out gold set rather than assumed to beat the best single parser.
  • I4 (learn): Evaluation reports should always give per-class or macro scores next to overall accuracy (for example, rare dependency relations or rare frame roles), so that wins on frequent categories cannot hide failures on rare ones.
  • I5 (tree): Hyperparameter sweeps for gparse could use the same sampler-plus-pruner setup, cutting GPU time while still exploring learning rate, dropout and sequence length.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of HoustonHouston · United States11 works · 1,210 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Bryan Tuck, Fatima Zahra Qachfar, Dainis Boumber and Rakesh Verma. 2023. DetectiveRedasers at ArAIEval Shared Task: Leveraging Transformer Ensembles for Arabic Deception Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.45

@inproceedings{tuck2023detectiveredasers,
  title = {{DetectiveRedasers at ArAIEval Shared Task: Leveraging Transformer Ensembles for Arabic Deception Detection}},
  author = {Bryan Tuck, Fatima Zahra Qachfar, Dainis Boumber and Rakesh Verma},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.45},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.45}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ArXiv preprint, abs/2102.10684. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework.
  4. Sarah Alqurashi, Ahmad Alhindi, and Eisa Alanazi. 2020. Large arabic twitter dataset on covid-19. ArXiv preprint, abs/2004.04315. ↗
  5. Samah M. Alzanin, Aqil M. Azmi, and Hatim A. Aboalsamh. 2022. Short text classification for Arabic social media tweets. Journal of King Saud University - Computer and Information Sciences, 34(9):6595– 6604. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  7. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. iCompass at Arabic hate speech 2022: Detect hate speech using QRNN and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180, Marseille, France. European Language Resources Association.
  8. Abdelali Bentahila and Eirlys E Davies. 1983. The syntax of arabic-french code-switching. Lingua, 59(4):301–330. ↗
  9. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 2546–2554. find
  10. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  11. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8:1–. ↗
  12. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  13. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  14. Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research, 18(185):1–52. find
  15. Ammar Mohammed and Rania Kora. 2023. A comprehensive review on ensemble deep learning: Opportunities and challenges. Journal of King Saud University - Computer and Information Sciences, 35(2):757– 774. ↗
  16. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023a. Detecting and identifying the reasons for deleted tweets before they are posted. Frontiers in Artificial Intelligence, 6. ↗
  17. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023b. Detecting and reasoning of deleted tweets before they are posted. ArXiv preprint, abs/2305.04927. ↗
  18. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2022. Emojis as anchors to detect arabic offensive language and hate speech. ↗
  19. Kai Shu, Amrita Bhattacharjee, Faisal Hammad Alatawi, Tahora H. Nazer, Kaize Ding, Mansooreh Karami, and Huan Liu. 2020. Combating disinformation in a social media age. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10. ↗
  20. Gadi Wolfsfeld, Elad Segev, and Tamir Sheafer. 2013. Social Media and the Arab Spring: Politics Comes First. The International Journal of Press/Politics, 18(2):115–137. Publisher: SAGE Publications Inc. ↗
  21. Zhi-Hua Zhou. 2012. Ensemble Methods: Foundations and Algorithms, 1st edition. Chapman & Hall/CRC. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuck-2023-detectiveredasers-araieval-shared-task-leveraging/