Elamur Research⌕
☰
home / papers / 2022 / SI2M & AIOX Labs at WANLP 2022 Shared Task: Propaganda Detection in Arabic, A Data Augmentation and Name Entity Recognition Approach

SI2M & AIOX Labs at WANLP 2022 Shared Task: Propaganda Detection in Arabic, A Data Augmentation and Name Entity Recognition Approach

Paper · 2022 · Kamel Gaanoun and Imade Benelallam

Summary

The authors tackle multi-label detection of propaganda techniques in Arabic tweets for the WANLP 2022 shared task. The task has 17 technique labels plus a "no technique" option, and training data is tiny and highly skewed. Their pipeline has four parts:

  • a span-recombination augmentation scheme called "MIX", which builds 2000 synthetic examples;
  • a fine-tuned ARBERT multi-label classifier whose decision threshold is tuned on Dev;
  • an AraBERT-based NER model, where any detected ORG entity triggers the "Name calling/Labeling" label;
  • a stopword-filtered rule that adds "Repetition" when repeated words appear.

An ablation on Dev shows each part helps, and NER contributes the largest jump. On the official Test set the system reached 0.585 micro F1 and placed 6th of 14 teams. A label-casing mismatch cost some points: with correct casing the score would have been 0.593. The work matters because Arabic propaganda detection has few resources. It also shows that cheap symbolic signals layered on a transformer can noticeably help a low-data, imbalanced task.

Key points

  • The data is small: 504 Train, 52 Dev and 323 Test sequences. "Loaded Language" (446) and "Name calling/Labeling" (244) dominate, and several techniques occur six times or fewer.

  • ARBERT was fine-tuned multi-label on a Tesla T4 for about 14 minutes:

    • 20 epochs, batch size 8, learning rate 5e-5, max length 512;
    • a threshold of 0.3 was picked by sweeping for the best Dev micro F1;
    • an empty prediction becomes "No technique".

  • Dev micro F1 rose at each step:

    ConfigurationDev micro F1
    Train only0.434
    + MIX0.455
    + Repetition0.459
    + NER (ORG)0.56

  • Official Test results were 0.585 micro F1 and 0.137 macro F1. The low macro score shows rare techniques remain largely unsolved.

  • Two variants did not help and were dropped:

    • using PER entities as a cue;
    • adding the English PTC corpus machine-translated into Arabic via Google Translate.

Interesting ideas

  • I1. Cut gold-annotated spans out of their sentences and splice random pairs into new multi-label examples, which multiplies scarce labelled data without any new annotation.
  • I2. Tune the multi-label probability threshold on held-out data, and fall back to an explicit "none" label when nothing clears it, instead of forcing a top-1 choice.
  • I3. Map a structural signal (an organization entity being present) directly to a rhetorical label; this simple rule gave a larger gain than the neural improvements.
  • I4. Let a deterministic rule only add a label the model missed, never remove one, so the symbolic component is a safe additive patch.
  • I5. A lost score from a capitalisation mismatch in a label string shows that output labels should be checked against the scorer's vocabulary before submission.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Verified silver clauses with agreed parses or frames could be recombined into synthetic multi-clause texts. This would stress-test training and coverage for rare constructions, ideally favouring rare labels as the authors suggest.
  • I2 (frames): Speech-act, irony or implicature detectors could use dev-calibrated thresholds with an explicit "no act/no irony" outcome. That keeps abstentions visible and explainable.
  • I3 (frames): Entity types from the world layer, such as an ORG or PER target, could serve as transparent features for spotting labeling, blame or praise in pragmatics. Each such feature would carry a traceable reason.
  • I4 (learn): Self-induced repair rules could be restricted to add-only edits on the parser or frame output. Their benefit could then be measured in isolation, as this paper's step-by-step ablation does.
  • I5 (learn): Evaluation harnesses could include a schema check that rejects unknown label strings, such as UD relations or frame names. Silent mismatches would then fail loudly rather than deflating results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Institut National de Statistique et d'Economie AppliquéeRabat · Morocco4 works · 64 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Kamel Gaanoun and Imade Benelallam. 2022. SI2M & AIOX Labs at WANLP 2022 Shared Task: Propaganda Detection in Arabic, A Data Augmentation and Name Entity Recognition Approach. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.58

@inproceedings{gaanoun2022si2m,
  title = {{SI2M \& AIOX Labs at WANLP 2022 Shared Task: Propaganda Detection in Arabic, A Data Augmentation and Name Entity Recognition Approach}},
  author = {Kamel Gaanoun and Imade Benelallam},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.58},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.58}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This paper describes our contribution to the shared task of propaganda detection in WANLP 2022. We propose a system based on data augmentation, Named Entity Recognition (NER), repetition detection, and ARBERT prediction for subtask 1 dealing with multi-label classification techniques. Our analysis shows that NER and data augmentation have a significant impact on the final results, placing us sixth out of 14 competing teams. *Table 4: Dev set results for each step* Step Wafa Saeed Murshid Al-Ziyadi. 2019. Propaganda-based Classification of Arabic Newspapers. Ph.D. thesis, Hamad Bin Khalifa University (Qatar). find
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  4. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. ↗
  5. Giovanni Da San Martino, Alberto Barron-Cedeno, and Preslav Nakov. 2019a. Findings of the nlp4if-2019 shared task on fine-grained propaganda detection. In Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda, pages 162–170. ↗
  6. Giovanni Da San Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the 14th Workshop on Semantic Evaluation, SemEval ’20, pages 1377–1414. ↗
  7. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5636–5646. find
  8. Gaku Morio, Terufumi Morishita, Hiroaki Ozaki, and Toshinori Miyoshi. 2020. Hitachi at semeval-2020 task 11: An empirical study of pre-trained transformer family for propaganda detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1739–1748. find
  9. Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Míguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani, et al. 2022. The clef-2022 checkthat! lab on fighting the covid-19 infodemic and fake news detection. In European Conference on Information Retrieval, pages 416–428. Springer. find
  10. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2931–2937.
  11. Abdulwahab Sahyoun. 2022. arabert-ner. https://huggingface.co/abdusahmbzuai/ arabert-ner. [Online; accessed 05-September-2022]. ↗
  12. Shehel Yoosuf and Yin Yang. 2019. Fine-grained propaganda detection with fine-tuned bert. In Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda, pages 87–91. find
  13. Kamel Gaanoun and Imade Benelallam. 2020. Arabic dialect identification: An arabic-bert model with data augmentation and ensembling strategy. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 275–281. ↗
  14. Manash Pratim Goswami. 2018. Fake news and cyber propaganda: A study of manipulation and abuses on social media. Mediascape in 21st Century: Emerging Perspectives, pages 535–544. find
  15. Institute for Propaganda Analysis. 1938. How to detect propaganda. Bulletin of the American Association of University Professors (1915-1955), 24(1):49–55. find
  16. Dawid Jurkiewicz, ukasz Borchmann, Izabela Kosmala, and Filip Graliński. 2020. Applicaai at semeval-2020 task 11: On roberta-crf, span cls and whether self-training helps them. arXiv preprint arXiv:2005.07934. ↗
  17. G Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. Semeval-2020 task 11: Detection of propaganda techniques in news articles. arXiv preprint arXiv:2009.02696. ↗
  18. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020b. A survey on computational propaganda detection. arXiv preprint arXiv:2007.08024. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gaanoun-2022-si2m-aiox-labs-wanlp-2022/