Elamur Research⌕
☰
home / papers / 2024 / Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach

Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach

Paper · 2024 · Md Abdur Razzaq Riyadh and Sara Nabhani

Summary

The paper describes a system for Task 1 of the ArAIEval 2024 shared task. The task asks a system to find which propaganda techniques (out of 23) appear in Arabic tweets and news text and to mark the exact character spans where they occur. The authors treat this as multi-label token tagging. Each subword token gets an independent yes/no decision for every technique, so labels can overlap.

They fine-tuned Multilingual BERT Cased (base) and tested which hidden layer should feed the classification head. They also trained AraBERTv2 as a monolingual reference, but only after the evaluation deadline. Hyperparameters were tuned with Optuna on short runs, and the model was then retrained for much longer.

The submitted mBERT system (10th layer) placed second in the shared task. Later experiments showed that the 8th layer worked better than the 10th or 12th, and that AraBERTv2 was stronger still. The work matters because it shows that choosing the right intermediate layer can narrow much of the gap between a multilingual model and an Arabic-specific one.

Key points

  • The data came from the organizers: 6997 training, 921 validation and 1046 test documents, about 30–34 tokens per document on average, annotated with 23 techniques that are unevenly distributed.
  • Each technique has a binary vector over the input, and runs of consecutive 1s form a span. Inputs were cut or padded to 256 subword tokens, and the binary cross-entropy loss ignored padding positions.
  • Hyperparameters were searched with Optuna (NSGA-II) over 5-epoch runs. The submitted system then trained for 400 epochs with batch size 32, a learning rate of about 6.91e-5 and 226 warmup steps, on one RTX A6000.
  • Test micro-F1 by layer:
    • mBERT layer 12: 0.24
    • mBERT layer 10: 0.28
    • mBERT layer 8: 0.33
    • AraBERTv2: 0.39
    • organizers' baseline: 0.01
  • Recall was well above precision (for example, mBERT layer 8 had 0.52 recall against 0.24 precision). Short spans of up to 3 words reached an F1 of about 0.10, while longer spans reached only about 0.02.

Interesting ideas

  • I1. The classifier head can be fed from an intermediate transformer layer rather than the top one, and here a lower layer worked best for span tagging. Layer choice is cheap to tune and can bring a sizeable gain.
  • I2. Spans are encoded as one independent binary track per label over the tokens. This lets several overlapping labels coexist without needing a BIO scheme for each combination.
  • I3. Errors are broken down by span length (short versus long), which exposed a large performance gap that a single score would have hidden.
  • I4. A qualitative review of false positives found several that looked plausible, which suggests the gold annotations are incomplete rather than the model being wrong.
  • I5. Hyperparameters are tuned with short, cheap runs, and only the winning configuration gets a long training run.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could compare arc and label scorers fed from different encoder layers, or from mixtures of them, instead of assuming the last layer is best.
  • I2 (frames): speech-act, irony or implicature cues that overlap the same words could be marked as parallel per-label tracks, each of which can be inspected separately.
  • I3 (learn): evaluation reports could routinely slice scores by span or constituent length, so that weaknesses on long arguments or clauses show up clearly.
  • I4 (learn): disagreements between witness parsers and the reference could be sampled and reviewed by hand before being counted as errors or turned into repair rules.
  • I5 (learn): this two-phase budget could speed up tuning of the perceptron parser or gparse without running full-length training for every candidate setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MaltaMsida · Malta29 works · 894 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Md Abdur Razzaq Riyadh and Sara Nabhani. 2024. Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.47

@inproceedings{riyadh2024mela,
  title = {{Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach}},
  author = {Md Abdur Razzaq Riyadh and Sara Nabhani},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.47},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.47}
}

References

18 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. Armeme: Propagandistic content in arabic memes.
  3. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  6. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2):182–197. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Khaldi Hadjer and Taqiy Bouklouha. 2023. HTE at ArAIEval shared task: Integrating content type information in binary persuasive technique detection. In Proceedings of ArabicNLP 2023, pages 502–507, Singapore (Hybrid). Association for Computational Linguistics. ↗
  9. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023a. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812. ↗
  10. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can gpt-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024, Torino, Italy. find
  11. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023b. Araieval shared task: Persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md.
  12. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. Araieval shared task: Propagandistic techniques detection in unimodal and multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  13. Ahmed Samir Hussein, Abu Bakr Soliman Mohammad, Mohamed Ibrahim, Laila Hesham Afify, and Samhaa R. El-Beltagy. 2022. NGU CNLP atWANLP 2022 shared task: Propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 545–550, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  14. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics.
  15. Shubham Mittal and Preslav Nakov. 2022. IITD at WANLP 2022 shared task: Multilingual multi-granularity network for propaganda detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 529–533, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  16. Olumide Ojo, Olaronke Adebanji, Hiram Calvo, Damian Dieke, Olumuyiwa Ojo, Seye Akinsanya, Tolulope Abiola, and Anna Feldman. 2023. Legend at ArAIEval shared task: Persuasion technique detection using a language-agnostic text representation model. In Proceedings of ArabicNLP 2023, pages 594–599, Singapore (Hybrid). Association for Computational Linguistics.
  17. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774. ↗
  18. Preprint, Fatima Zahra Qachfar and Rakesh Verma. 2023. ReDASPersuasion at ArAIEval shared task: Multilingual and monolingual models for Arabic persuasion detection. In Proceedings of ArabicNLP 2023, pages 549–557, Singapore (Hybrid). Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riyadh-2024-mela-araieval-shared-task-propagandistic/