Elamur Research⌕
☰
home / papers / 2023 / ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text

ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text

Paper · 2023 · Maram Hasanain, Firoj Alam, Hamdy Mubarak et al.

Summary

ArAIEval, run with ArabicNLP 2023 (co-located with EMNLP 2023), is a shared task on two kinds of manipulative Arabic content. Task 1 asks systems to recognise persuasion techniques in short multi-genre snippets: tweets from news-outlet accounts and paragraphs from news pages linked to fact-checked claims in AraFacts. It has a binary subtask (1A, whether any technique is present) and a multilabel subtask (1B, which of 23 techniques are used). Task 2 covers disinformation in tweets, in binary and multiclass versions. Three annotators labelled each snippet independently, and two consolidators then merged their work into the gold labels. Twenty teams submitted to the final evaluation. Most of them fine-tuned Arabic pretrained transformers such as AraBERT or MARBERT. Some also experimented with loss functions, data augmentation or ensembles. The organisers note that Task 2 scores were very high and suggest the dataset may have been too easy. All data and scoring scripts are public, which gives Arabic propaganda and disinformation research a reusable benchmark.

Key points

  • Participation: 20 teams in the final phase (14 on Task 1, 16 on Task 2), and 17 system description papers.
  • Task 1 has 2427 train, 259 dev and 503 test snippets. Snippets with at least one technique make up 79%, 78% and 66% of these splits.
  • The label set has 23 persuasion techniques taken from Piskorski et al. (2023). Loaded Language is by far the most frequent, with 1574 train instances.
  • The news paragraphs come from AraFacts pages on news domains, split at blank lines. The 156 test tweets were newly annotated, while the train and dev tweets reuse the WANLP 2022 propaganda data.
  • System approaches:
    • Raphael added GPT-3.5 English descriptions encoded with RoBERTa.
    • rematchka used Arabic–English back-translation and asymmetric loss.
    • UL & UM6P used RegMixup and Focal Tversky losses.
    • USTHB used SVM on FastText and TF-IDF features.

Interesting ideas

  • I1. The binary label is computed from the fine-grained multilabel annotation (true if any technique is present), so the two subtasks can never contradict each other.
  • I2. Gold labels come from three independent annotators plus two consolidators who reconcile them, rather than from a simple majority vote.
  • I3. Splits are stratified by technique so that rare labels appear in train, dev and test in similar proportions.
  • I4. The organisers read unusually high multiclass scores as a sign that the dataset may be too easy, not as proof that the task is solved.
  • I5. One team had an LLM write an English description of each Arabic text, including its tone, and fed that description in as an extra feature view.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): coarse flags in prag, such as "contains irony" or "is an indirect request", could be derived mechanically from the detailed frame annotations instead of being labelled separately.
  • I2 (learn): where witness parsers disagree, a separate adjudication pass could resolve the conflict and record its reasons, which keeps silver data explainable.
  • I3 (learn): evaluation sets for rare constructions or frame types should be stratified the same way, so that a missing class does not distort the scores.
  • I4 (learn): suspiciously strong results could automatically trigger negative-control checks, such as shuffled labels or shallow baselines, to detect shortcut cues.
  • I5 (learn): LLM teacher tiers could supply short explanatory glosses alongside labels, giving the system a readable rationale to check its own frame or stance decisions against.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓University of TrentoTrento · Italy85 works · 15,524 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.44

@inproceedings{hasanain2023araieval,
  title = {{ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text}},
  author = {Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni {Da San Martino} and Abed Alhakim Freihat},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.44},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.44}
}

References

58 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem Abdel-Salam. 2023. rematchka at ArAIEval Shared Task: Prefix-Tuning & Prompt-tuning for Improved Detection of Propaganda and Disinformation in Arabic Social Media Content. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. ↗
  2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  4. Nouman Ahmed, Natalia Flechas Manrique, and Jehad Oumer. 2023. Itri Amigos at ArAIEval Shared Task: Transformer vs. Compression-Based Models for Persuasion Techniques and Disinformation Detection. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  5. Wahhab Ahmed Bahaaulddin A., Sabeeh Vian, Belhaj Hanan Mohamed, Sibaee Serry, Samar Ahmad, Khurfan Ibrahim, and Alharbi Abdullah I. 2023. AraDetector at ArAIEval Shared Task: An Ensemble of Arabic-specific pre-trained BERT and GPT-4 for Arabic Disinformation Detection . In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  6. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  7. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022b. find
  8. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, WANLP ’22, Abu Dhabi, UAE.
  9. Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, Abdulaziz Al-Homaid, Wajdi Zaghouani, Tommaso Caselli, Gijs Danoe, Friso Stolk, Britt Bruntink, and Preslav Nakov. 2021. Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 611–649, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  10. Zien Sheikh Ali, Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2021. AraFacts: the first large arabic dataset of naturally occurring claims. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 231–236.
  11. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  13. Dilshod Azizov. 2023. Frank at ArAIEval Shared Task: Arabic Disinformation and Persuasion: Power of Pre-trained Models. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore. Association for Computational Linguistics. find
  14. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. ↗
  15. Zakaria Boulouard, Mariya Ouaissa, Mariyam Ouaissa, Moez Krichen, Mutiq Almutiq, and Gasmi Karim. 2022. Detecting hateful and offensive speech in arabic social media using transfer learning. Applied Sciences, 12:12823. ↗
  16. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  17. Ralph D. Casey. 1994. What is propaganda? historians.org. find
  18. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. find
  19. Giovanni Da San Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. Semeval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the fourteenth workshop on semantic evaluation, pages 1377–1414. ↗
  20. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, EMNLP-IJCNLP 2019, Hong Kong, China. ↗
  21. Pritam Deka and Ashwathy Revi. 2023. PD-AR at ArAIEval Shared Task: Persuasion techniques detection: an interdisciplinary approach to identifying and counteracting manipulative strategies. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. find
  22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  23. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. ↗
  24. Ahmed ElSayed, Omar Nasr, and Nour Eldin Elmadany. 2023. AAST-NLP at ArAIEval Shared Task: Tackling Persuasion Technique and Disinformation Detection using Pre-Trained Language Models On Imbalanced Datasets. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore. Association for Computational Linguistics.
  25. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1–30. ↗
  26. Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7–12, Copenhagen, Denmark. Association for Computational Linguistics. find
  27. Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018. Adapting Serious Game for Fallacious Argumentation to German: Pitfalls, Insights, and Best Practices. find
  28. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 3329–3335. find
  29. Maram Hasanain, Ahmed El-Shangiti, Rabindra Nath Nandi, Preslav Nakov, and Firoj Alam. 2023. QCRI at SemEval-2023 task 3: News genre, framing and persuasion techniques detection using multilingual models. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1237–1244, Toronto, Canada. Association for Computational Linguistics. find
  30. Fatemah Husain and Ozlem Uzuner. 2021. A survey of offensive language detection for the arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1):1–44. ↗
  31. Cherilyn Ireton and Julie Posetti. 2018. Journalism, fake news & disinformation: handbook for journalism education and training. Unesco Publishing. find
  32. Areej Jaber and Paloma Martinez. 2023. PTUK-HULAT at ArAIEval Shared Task: Fine-tuned Distilbert to Predict Disinformative Tweets. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. ↗
  33. Garth S Jowett and Victoria O’donnell. 2018. Propaganda & persuasion. Sage publications. find
  34. Hadjer Khaldi and Taqiy Eddine Bouklouha. 2023. HTE at ArAIEval Shared Task: Persuasion techniques detection: an interdisciplinary approach to identifying and counteracting manipulative strategies. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. find
  35. Salima Lamsiyah, El Mahdaouy Abdelkader, Hamza Alami, Ismail Berrada, and Christoph Schommer. 2023. UL& UM6P at ArAIEval Shared Task: Transformer-based model for Persuasion Techniques and Disinformation detection in Arabic. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  37. Sudeep Mangalvedhekar, Kshitij Deshpande, Yash Patwardhan, Vedant Deshpande, and Ravindra Mu-rumkar. 2023. Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  38. V. Margolin. 1979. The visual rhetoric of propaganda. Information Design Journal, 1:107–122. ↗
  39. Priyanka Meel and Dinesh Kumar Vishwakarma. 2020. Fake news, rumor, information pollution in social media and web: A contemporary survey of state-of-the-arts, challenges and opportunities. Expert Systems with Applications, 153:112986. ↗
  40. Clyde R. Miller. 1939. The Techniques of Propaganda. From “How to Detect and Analyze Propaganda,” an address given at Town Hall. The Center for learning. find
  41. Lichouri Mohamed, Lounnas Khaled, Zitouni Aicha, Latrache Houda, and Djeradi Rachida. 2023. USTHB at ArAIEval Shared Task: Disinformation Detection System based on Linguistic Feature Concatenation. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. find
  42. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  43. Hamdy Mubarak, Ahmed Abdelali, Sabit Hassan, and Kareem Darwish. 2020a. Spam detection on arabic twitter. In Social Informatics: 12th International Conference, SocInfo 2020, Pisa, Italy, October 6–9, 2020, Proceedings 12, pages 237–251. Springer. find
  44. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020b. Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection, pages 48–52.
  45. Hamdy Mubarak, Sabit Hassan, Shammur Absar Chowdhury, and Firoj Alam. 2022. ArCovidVac: Analyzing Arabic tweets about COVID-19 vaccination. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3220–3230, Marseille, France. European Language Resources Association.
  46. Olumide E. Ojo, Olaronke O. Adebanji, Hiram Calvo, Damian O. Dieke, Olumuyiwa E. Ojo, Seye E. Akin-sanya, Tolulope O. Abiola, and Anna Feldman. 2023. Legend at ArAIEval Shared Task: Persuasion Technique Detection using a Language-Agnostic Text Representation Model. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  47. OpenAI. 2023. GPT-4 technical report. Technical report, OpenAI. ↗
  48. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361, Toronto, Canada. Association for Computational Linguistics. find
  49. Fatima Zahra Qachfar and Rakesh M. Verma. 2023. ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection. In Proceedings of the First Arabic Natural Language Processing Conference (Arabic-NLP 2023), Singapore. Association for Computational Linguistics.
  50. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  51. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Shivam Sharma, Firoj Alam, Md. ↗
  52. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI ’22, pages 5597–5606, Vienna, Austria. International Joint Conferences on Artificial Intelligence Organization. Survey Track. find
  53. Bryan E. Tuck, Fatima Zahra Qachfar, Dainis Boumber, and Rakesh M. Verma. 2023. DetectiveRedasers at ArAIEval Shared Task: Leveraging Transformer Ensembles for Arabic Deception Detection. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  54. Shukla Utsav, Tiwari Shailendra, and Vyas Manan. 2023. Raphael at ArAIEval Shared Task: Understanding Persuasive Language and Tone, an LLM Approach. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  55. Hariram Veeramani, Surendrabikram Thapa, and Usman Naseem. 2023. KnowTellConvince at ArAIEval Shared Task: Disinformation and Persuasion Detection using Similar and Contrastive Representation Alignment. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. find
  56. Yunze Xiao and Firoj Alam. 2023. Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detection. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. ↗
  57. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235. ↗
  58. Lu Zhou, Wenbo Wang, and Keke Chen. 2016. Tweet properly: Analyzing deleted tweets to understand and identify regrettable ones. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 603–612, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hasanain-2023-araieval-shared-task-persuasion-techniques/