Elamur Research⌕
☰
home / papers / 2023 / AraDetector at ArAIEval Shared Task: An Ensemble of Arabic-specific pre-trained BERT and GPT-4 for Arabic Disinformation Detection

AraDetector at ArAIEval Shared Task: An Ensemble of Arabic-specific pre-trained BERT and GPT-4 for Arabic Disinformation Detection

Paper · 2023 · Ahmed Bahaaulddin, Vian Sabeeh, Hanan Belhaj et al.

Summary

This paper describes a system entered in Task 2 (disinformation detection) of the ArAIEval shared task at ArabicNLP 2023. The task is a binary decision: is an Arabic tweet disinformation or not. The organizers supplied a Twitter dataset with 14,126 training tweets and 2115 development tweets, scored by micro-F1.

The authors first cleaned and normalized the text. They then fine-tuned three Arabic encoders, AraBERT, QARiB and MARBERT, and compared them with GPT-4 used in two ways:

  • zero-shot, with a prompt casting the model as an expert annotator;
  • few-shot, with nine training tweets chosen to be as different from each other as possible.

The best encoder (MARBERT), zero-shot GPT-4 and few-shot GPT-4 were combined by majority vote. On the dev set, zero-shot GPT-4 scored highest (79.01% micro-F1), and the ensemble reached 76.83%. The team still submitted the ensemble, which scored 74.87% micro-F1 on the official test set.

The work shows that a prompted LLM can beat fine-tuned Arabic encoders on this task. It also shows that the aggregate score can hide large differences in how well each system catches the rare class.

Key points

  • Preprocessing: non-Arabic letters, punctuation and diacritics were removed, while emojis were kept. Hamza variants and ta marbuta were normalized to single forms. This helped every encoder; MARBERT gained the most (1.52%) and QARiB the least (0.18%).
  • Fine-tuned encoders (dev micro-F1 with preprocessing): MARBERT 69.79%, QARiB 69.22%, AraBERT 68.64%.
  • GPT-4 on dev: zero-shot reached 79.01% micro-F1, while few-shot dropped to 55.74%. However, few-shot found 337 of the 397 disinformation tweets, compared with only 168 for zero-shot.
  • Few-shot example selection:
    • tweets were embedded with the sentence-transformers model distiluse-base-multilingual-cased-v2;
    • after a random first pick, each new example was the one most distant (by cosine) from those already chosen;
    • the minority class was deliberately overweighted: 6 disinformation and 3 non-disinformation examples.
  • Ensemble results: the majority vote of MARBERT, zero-shot and few-shot scored 76.83% micro-F1 and 64.43% macro-F1 on dev. On test it scored 74.87% micro-F1 and 64.98% macro-F1.

Interesting ideas

  • I1. Choosing in-context examples by repeatedly taking the item farthest from those already chosen gives a small but varied prompt set instead of a random handful
  • I2. Deliberately putting more minority-class examples into the prompt raised recall on the rare class sharply, even though overall micro-F1 fell
  • I3. Reporting per-class hits next to micro-F1 exposed that the top-scoring setup missed most of the positive class
  • I4. Majority voting over very different classifiers (a fine-tuned encoder plus two prompting styles) aims to cancel out each one's typical errors
  • I5. Normalization merged spelling variants but kept emojis because they carry meaning, so cleaning was selective rather than aggressive

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could pick their demonstration sentences this way from a parse-embedding space, so a few examples cover more distinct constructions.
  • I2 (learn): when prompting a teacher for rare phenomena (irony, unusual frames), overweighting them in the demonstrations could raise recall, with an evaluation step guarding precision.
  • I3 (learn): evaluation reports should break results down by rare labels and constructions, so a high aggregate cannot hide a blind spot.
  • I4 (learn): silver-data agreement could draw witnesses from different families (perceptron parser, gparse, LLM teacher) and keep only items where most agree.
  • I5 (tokens): the tokenizer could normalize orthographic variants while keeping emoji and similar signal-bearing tokens as their own units for later pragmatic use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations🎓Libyan AcademyTripoli · Libya2 works · 7 citations🎓Middle Technical UniversityBaghdad · Iraq2 works · 7 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Bahaaulddin, Vian Sabeeh, Hanan Belhaj, Serry Sibaee, Samar Ahmad, Ibrahim Khurfan and Abdullah Alharbi. 2023. AraDetector at ArAIEval Shared Task: An Ensemble of Arabic-specific pre-trained BERT and GPT-4 for Arabic Disinformation Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.51

@inproceedings{bahaaulddin2023aradetector,
  title = {{AraDetector at ArAIEval Shared Task: An Ensemble of Arabic-specific pre-trained BERT and GPT-4 for Arabic Disinformation Detection}},
  author = {Ahmed Bahaaulddin, Vian Sabeeh, Hanan Belhaj, Serry Sibaee, Samar Ahmad, Ibrahim Khurfan and Abdullah Alharbi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.51},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.51}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, et al. 2021. Arbert & marbert: Deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105. ↗
  3. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics.
  4. Maha Al-Yahya, Hend Al-Khalifa, Heyam Al-Baity, Duaa AlSaeed, and Amr Essam. 2021. Arabic Fake News Detection: Comparative Study of Neural Networks and Transformer-Based Approaches. Complexity, 2021:5516945. Publisher: Hindawi. ↗
  5. Naelah O Bahurmuz, Ghada A Amoudi, Fatmah A Baothman, Amani T Jamal, Hanan S Alghamdi, and Areej M Alhothali. 2022. Arabic Rumor Detection Using Contextual Deep Bidirectional Language Modeling. IEEE Access, 10:114907–114918. Publisher: IEEE. ↗
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis. Association for Computational Linguistics.
  8. Israel Doncel-Martín, Daniel Catalan-Matamoros, and Carlos Elías. 2023. Corporate social responsibility and public diplomacy as formulas to reduce hate speech on social media in the fake news era. Corporate Communications: An International Journal, 28(2):340–352. Publisher: Emerald Publishing Limited. ↗
  9. Zaid Alyafeai, Maged S Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating arabic nlp tasks using chatgpt models. arXiv preprint arXiv:2306.16322. ↗
  10. Rehab Duwairi and Mahmoud El-Orfali. 2014. A study of the effects of preprocessing strategies on sentiment analysis for Arabic text. Journal of Information Science, 40(4):501–513. ↗
  11. Mohamed Seghir Hadj Ameur and Hassina Aliane. 2021. AraCOVID19-MFH: Arabic COVID-19 Multi-Deen Freelon and Chris Wells. 2020. Disinformation as political communication. find
  12. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abdelhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  13. Hanen Himdi, George Weir, Fatmah Assiri, and Hassanin Al-Barhamtoshy. 2022. Arabic fake news detection based on textual analysis. Arabian Journal for Science and Engineering, 47(8):10453–10469. ↗
  14. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, page 1–22. ↗
  15. Rasha Obeidat, Maram Gharaibeh, Malak Abdullah, and Yara Alharahsheh. 2022. Multi-label multi-class COVID-19 Arabic Twitter dataset with fine-grained misinformation and situational information annotations. PeerJ Computer Science, 8:e1151. ↗
  16. OpenAI. 2023. Gpt-4 technical report.
  17. R. M. Albalawi, A. T. Jamal, A. O. Khadidos, and A. M. Alhothali. 2023. Multimodal Arabic Rumors Detection. IEEE Access, 11:9716–9730. ↗
  18. Kai Shu, Amrita Bhattacharjee, Faisal Alatawi, Tahora H Nazer, Kaize Ding, Mansooreh Karami, and Huan Liu. 2020. Combating disinformation in a social media age. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(6):e1385. ↗
  19. Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023. Is chatgpt a good sentiment analyzer? a preliminary study. ↗
  20. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848. ↗
  21. Han Zheng, Jinhui Li, Charles T. Salmon, and Yin-Leng Theng. 2020. The effects of exergames on emotional well-being of older adults. Computers in Human Behavior, 110:106383. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bahaaulddin-2023-aradetector-araieval-shared-task-ensemble/