Elamur Research⌕
☰
home / papers / 2023 / Frank at ArAIEval Shared Task: Arabic Persuasion and Disinformation: The Power of Pretrained Models

Frank at ArAIEval Shared Task: Arabic Persuasion and Disinformation: The Power of Pretrained Models

Paper · 2023 · Dilshod Azizov, Jiyong Li and Shangsong Liang

Summary

This system paper reports one team's entries to the ArAIEval shared task at ArabicNLP 2023. The task covered two binary problems on Arabic text: Subtask 1A asks whether a snippet (a tweet or a news paragraph) uses a persuasion technique, and Subtask 2A asks whether a tweet is disinformation. Both datasets are small and strongly skewed toward one class.

The authors fine-tuned several pretrained encoders and kept the best one for each subtask. For model selection they held out 10% of the training data and treated the official dev set as a local test set. mBERT won for persuasion and MARBERT won for disinformation. Both used the AraBERT preprocessor with inputs capped at 512 tokens.

On the official leaderboard the persuasion system reached micro-F1 0.745 and placed 7th of 16. The disinformation system reached micro-F1 0.816 and placed 12th of 17. Its macro-F1 was much lower, at 0.637.

The work matters mainly as a baseline. It shows how far plain fine-tuning of off-the-shelf encoders goes on Arabic persuasion and disinformation detection, which are still under-resourced tasks.

Key points

  • Data sizes: Subtask 1A has about 3.2k items (2,120 train / 566 dev / 503 test). Subtask 2A has about 20k tweets (14,147 / 2,115 / 3,739). In the train and dev sets, persuasion ("True") is the large majority, while "Disinfo" is the minority class.
  • Models compared:
    • Subtask 1A: mBERT, XLM-RoBERTa and MARBERT, with dev micro-F1 of 0.889, 0.876 and 0.881.
    • Subtask 2A: MARBERT, ALBERT and mBERT, with dev micro-F1 of 0.866, 0.846 and 0.840.
  • Final settings:
    • mBERT: batch 16, 5 epochs, lr 5e-5, AdamW with weight decay 0.01, gradient clipping at 1.0, no warm-up.
    • MARBERT (163M parameters): Adam, lr 5e-5, batch 32, 5 epochs.
  • Official results:
    • Subtask 1A: micro-F1 0.745 and macro-F1 0.717, 13.2% above the baseline.
    • Subtask 2A: micro-F1 0.816 and macro-F1 0.637, 6.7% above the majority-class baseline.
  • Future work: the authors propose adding meta-features of the text, such as length, unique word count and sentiment.

Interesting ideas

  • I1. Reporting macro-F1 next to micro-F1 exposed a weakness: on the skewed disinformation data, the drop from 0.816 to 0.637 shows the minority class is handled poorly, which the official micro score alone would hide.
  • I2. Results were stated as a margin over a naive majority-class predictor, which shows how much the model actually learned beyond the label skew.
  • I3. The authors carved their own dev split out of training data and kept the official dev set as an untouched test, so choosing a model did not peek at the final evaluation set.
  • I4. No single encoder won both tasks: the multilingual mBERT was best for mixed-genre persuasion, while the tweet-trained MARBERT was best for tweet disinformation, so the domain of pretraining data mattered per task.
  • I5. The planned shallow meta-features (length, distinct-word count, sentiment) are cheap, interpretable signals that could sit alongside a neural encoder.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every evaluation should report per-class or macro scores for rare labels (e.g. rare dependency relations or speech acts), so that progress on frequent cases cannot mask failures on rare ones.
  • I2 (learn): Include a majority-label baseline as a standard negative control in every classifier evaluation, such as irony or speech-act tagging.
  • I3 (learn): When choosing among witness parsers or repair rules, tune on a split taken from training data and keep the gold dev set untouched for the final check.
  • I4 (tree): Select the gparse encoder or treebank mix per text genre (news, fables, school texts) instead of assuming one model is best everywhere.
  • I5 (frames): Simple explainable features such as sentence length and lexical variety could serve as transparent extra evidence in irony or persuasion-style judgments in prag.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China72 works · 9,502 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Dilshod Azizov, Jiyong Li and Shangsong Liang. 2023. Frank at ArAIEval Shared Task: Arabic Persuasion and Disinformation: The Power of Pretrained Models. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.59

@inproceedings{azizov2023frank,
  title = {{Frank at ArAIEval Shared Task: Arabic Persuasion and Disinformation: The Power of Pretrained Models}},
  author = {Dilshod Azizov, Jiyong Li and Shangsong Liang},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.59},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.59}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Nawaf A Abdulla, Nizar A Ahmed, Mohammed A Shehab, and Mahmoud Al-Ayyoub. 2013. Arabic sentiment analysis: Lexicon-based and corpus-based. In 2013 IEEE Jordan conference on applied electrical engineering and computing technologies (AEECT), pages 1–6. IEEE. find
  3. Ahmed Al-Rawi, Abdelrahman Fakida, Kelly Grounds, et al. 2022. Investigation of covid-19 misinformation in arabic on twitter: Content analysis. Jmir Infodemiology, 2(2):e37007. ↗
  4. Ahmad Al-Sallab, Ramy Baly, Hazem Hajj, Khaled Bashir Shaban, Wassim El-Hajj, and Gilbert Badaro. 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):1–20. find
  5. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  6. Ali Almiman, Nada Osman, and Marwan Torki. 2020. Deep neural network approach for arabic community question answering. Alexandria Engineering Journal, 59(6):4427–4434. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  8. Dilshod Azizov, S Liang, and P Nakov. 2023. Frank at checkthat! 2023: Detecting the political bias of news articles and news media. Working Notes of CLEF. find
  9. Alberto Barrón-Cedeño, Firoj Alam, Tommaso Caselli, Giovanni Da San Martino, Tamer Elsayed, Andrea Galassi, Fatima Haouari, Federico Ruggeri, Julia Maria Struß, Rabindra Nath Nandi, et al. 2023. The clef-2023 checkthat! lab: Checkworthiness, subjectivity, political bias, factuality, and authority. In European Conference on Information Retrieval, pages 506–517. Springer. find
  10. Alberto Barrón-Cedeño, Firoj Alam, Andrea Galassi, Giovanni Da San Martino, Preslav Nakov, , Tamer Elsayed, Dilshod Azizov, Tommaso Caselli, Gullal S. Cheema, Fatima Haouari, Maram Hasanain, Mucahid Kutlu, Chengkai Li, Federico Ruggeri, Julia Maria Struß, and Wajdi Zaghouani. 2023. Overview of the CLEF–2023 CheckThat! Lab checkworthiness, subjectivity, political bias, factuality, and authority of news articles and their source. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the CLEF Association (CLEF 2023). find
  11. Alberto Barrón-Cedeño, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, Shaden Shaar, and Zien Sheikh Ali. Overview of CheckThat! 2020: Automatic identification and verification of claims in social media. pages 215–236. find
  12. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831. find
  13. Giovanni Da San Martino, Firoj Alam, Maram Hasanain, Rabindra Nath Nandi, Dilshod Azizov, and Preslav Nakov. 2023. Overview of the clef-2023 checkthat! lab task 3 on political bias of news articles and news media. Working Notes of CLEF. find
  14. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5636–5646. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  16. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Semeval-2021 task 6: detection of persuasion techniques in texts and images. arXiv preprint arXiv:2105.09284. ↗
  17. Tamer Elsayed, Preslav Nakov, Alberto Barrón-Cedeño, Maram Hasanain, Reem Suwaileh, Giovanni Da San Martino, and Pepa Atanasova. 2019a. Check-That! at CLEF 2019: Automatic identification and verification of claims. In Advances in Information Retrieval – European Conference on IR Research, ECIR ’19, pages 309–315. find
  18. Tamer Elsayed, Preslav Nakov, Alberto Barrón-Cedeño, Maram Hasanain, Reem Suwaileh, Giovanni Da San Martino, and Pepa Atanasova. 2019b. Overview of the CLEF-2019 CheckThat!: Automatic identification and verification of claims. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, LNCS, pages 301–321.
  19. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507.
  20. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2020. Arcov19-rumors: Arabic covid-19 twitter dataset for misinformation detection. arXiv preprint arXiv:2010.08768. ↗
  21. Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021. A survey on stance detection for mis-and disinformation identification. arXiv preprint arXiv:2103.00242. ↗
  22. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abdelhakim Freihat. 2023a. Araieval shared task: Persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  23. Maram Hasanain, Ahmed Oumar El-Shangiti, Rabindra Nath Nandi, Preslav Nakov, and Firoj Alam. 2023b. Qcri at semeval-2023 task 3: News genre, framing and persuasion techniques detection using multilingual models. arXiv preprint arXiv:2305.03336. ↗
  24. Maram Hasanain, Fatima Haouari, Reem Suwaileh, Zien Sheikh Ali, Bayan Hamdan, Tamer Elsayed, Alberto Barrón-Cedeño, Giovanni Da San Martino, and Preslav Nakov. Overview of CheckThat! 2020 Arabic: Automatic identification and verification of claims in social media. find
  25. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  27. G Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. Semeval-2020 task 11: Detection of propaganda techniques in news articles. arXiv preprint arXiv:2009.02696. ↗
  28. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020b. A survey on computational propaganda detection. arXiv preprint arXiv:2007.08024. ↗
  29. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  30. Preslav Nakov, Firoj Alam, Giovanni Da San Martino, Maram Hasanain, RN Nandi, D Azizov, and P Panayotov. 2023. Overview of the clef-2023 checkthat! lab task 4 on factuality of reporting of news media. Working Notes of CLEF. find
  31. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361. find
  32. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A survey on arabic named entity recognition: Past, recent advances, and future trends. arXiv preprint arXiv:2302.03512. ↗
  33. Adharsh Raj and Manash Pratim Goswami. 2020. Is fake news spreading more rapidly than covid-19 in india. Journal of Content, Community and Communication, 11(10):208–220. find
  34. Samuel C Woolley and Philip N Howard. 2018. Computational propaganda: Political parties, politicians, and political manipulation on social media. Oxford University Press. find
  35. Qiang Zhang, Aldo Lipani, Shangsong Liang, and Emine Yilmaz. 2019. Reply-aided detection of misinformation via bayesian deep learning. In The world wide web conference, pages 2333–2343. ↗
  36. Preslav Nakov and Giovanni Da San Martino. 2021. Fake news, disinformation, propaganda, and media bias. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 4862–4865. find
  37. Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021a. The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Advances in Information Retrieval – European Conference on IR Research, pages 639–649. find
  38. Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021b. The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Advances in Information Retrieval – 43rd European Conference on IR Research, volume 12657 of ECIR ’21, pages 639– 649. find
  39. Wei Peng, Sue Lim, and Jingbo Meng. 2023. Persuasive strategies in online health misinformation: a systematic review. Information, Communication & Society, 26(11):2131–2148. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/azizov-2023-frank-araieval-shared-task-arabic/