Elamur Research⌕
☰
home / papers / 2023 / ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection

ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection

Paper · 2023 · Fatima Zahra Qachfar and Rakesh Verma

Summary

The paper describes a University of Houston entry to the ArAIEval 2023 shared task on spotting persuasion techniques in Arabic text. There are two subtasks: Task 1A is binary (does the text contain persuasion or not), and Task 1B is multi-label (which of 23 techniques are present). The data combines Modern Standard Arabic news paragraphs with tweets in many dialects.

The authors adapt ReDASPersuasion, their earlier multilingual system that joins a transformer encoder with language-agnostic persuasion features. They run 22 experiments covering classical TF-IDF baselines, multilingual transformers and Arabic monolingual transformers. Tweets are cleaned first: platform tokens are rewritten in Arabic, foreign-language fragments are translated, and emojis become Arabic descriptions.

On the test set, XLM-RoBERTa is the best encoder for the binary subtask. The dialect-tuned CAMeLBERT-DA SA wins on micro-F1 for the multi-label subtask. The takeaway is that the best encoder depends on the subtask. Macro scores stay very low because many techniques are rare.

Key points

  • Task 1A test results: ReDASPersuasion with XLM-RoBERTa reaches F1-micro 0.7336 (on dev it scored 0.8764).
  • Task 1B test results:
    • CAMeLBERT-DA SA gives the best F1-micro, 0.5584.
    • XLM-RoBERTa gives the best F1-macro, 0.1449.
  • Labels: The 23 Task 1B techniques fall into six groups, including Justification, Simplification, Distraction and Attack on reputation. "Appeal to Popularity" appears only twice in training and gets an F1 of zero.
  • Degenerate models and a wrong submission: On 1A, LR, the majority-class baseline and DistilBERT all score exactly 0.6581, meaning each labels everything "true". The team mistakenly submitted the DistilBERT predictions instead of the XLM-RoBERTa ones.
  • Preprocessing tools:
    • Lingua detects code-switched spans, which Google's translation API then translates.
    • The emoji package was extended with Arabic names taken from Unicode CLDR annotations.
    • Baselines are tuned with HyperOpt.

Interesting ideas

  • I1. Instead of deleting emojis, the authors replace each one with its descriptive name in the text's language, so the emotional signal stays in a form the model can read.
  • I2. To prevent outputs like "no technique plus Loaded Language", the "no technique" class is encoded as an empty label set rather than a competing label.
  • I3. Several unrelated models posting exactly the majority-class score revealed that they had collapsed into predicting a single label.
  • I4. Reporting micro and macro F1 side by side showed that strong micro scores can hide near-zero performance on rare classes.
  • I5. Platform tokens such as RT, @USER and LINK were rewritten as ordinary words in the main language before modelling, making noisy social text look more like regular prose.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tokenizer could map emojis to lexical items via CLDR English annotations, giving them lemmas and POS so later stages can treat them as sentiment-bearing words.
  • I2 (frames): Absence of irony or of a speech act should be modelled as an empty set of detected phenomena, not as a rival class that can co-fire with real ones.
  • I3 (learn): Every evaluation should include a majority-class control and automatically flag any model whose score matches it, as a cheap negative control.
  • I4 (learn): Frame and role classifiers should be reported with macro or per-class scores so rare VerbNet roles or speech acts are not masked by frequent ones.
  • I5 (tokens): Normalising handles, URLs and retweet markers into placeholder nouns before tagging would let the UD parser attach them sensibly rather than producing broken trees.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of HoustonHouston · United States11 works · 1,210 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Fatima Zahra Qachfar and Rakesh Verma. 2023. ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.54

@inproceedings{qachfar2023redaspersuasion,
  title = {{ReDASPersuasion at ArAIEval Shared Task: Multilingual and Monolingual Models For Arabic Persuasion Detection}},
  author = {Fatima Zahra Qachfar and Rakesh Verma},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.54},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.54}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849– 1864. ↗
  5. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. icompass at arabic hate speech 2022: Detect hate speech using qrnn and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180.
  6. James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115–123. PMLR. find
  7. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  8. Bechir Brahem, Hatem Haddad, et al. 2022. icompass at wanlp 2022 shared task: Arbert and marbert for multilabel propaganda classification of arabic tweets. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 511– 514.
  9. Leo Breiman. 2001. Random forests. Machine learning, 45:5–32. find
  10. Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. 2015. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1–4. find
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. find
  12. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20:273–297. ↗
  13. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. Ukraine (Online). Association for Computational Linguistics. find
  14. Barbara Johnstone Koch. 1983. Presentation as proof: The language of arabic rhetoric. Anthropological linguistics, pages 47–60. find
  15. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. ↗
  16. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2022. Emojis as anchors to detect arabic offensive language and hate speech. ↗
  17. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
  19. Fatima Zahra Qachfar and Rakesh Verma. 2023. ReDASPersuasion at SemEval-2023 task 3: Persuasion detection using multilingual transformers and language agnostic features. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2124–2132, Toronto, Canada. Association for Computational Linguistics. find
  20. E.S. Glenn, D. Witmeyer, and K.A. Stevenson. 1977. Cultural styles of persuasion. International Journal of Intercultural Relations, 1(3):52–66. ↗
  21. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  22. Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7–12, Copenhagen, Denmark. Association for Computational Linguistics. find
  23. Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018. Adapting serious game for fallacious argumentation to german: Pitfalls, insights, and best practices. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), page in press, Miyazaki, Japan. European Language Resources Association (ELRA). find
  24. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  25. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Raymond E Wright. 1995. Logistic regression.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qachfar-2023-redaspersuasion-araieval-shared-task-multilingual/