Elamur Research⌕
☰
home / papers / 2022 / NGU CNLP atWANLP 2022 Shared Task: Propaganda Detection in Arabic

NGU CNLP atWANLP 2022 Shared Task: Propaganda Detection in Arabic

Paper · 2022 · Ahmed Samir Hussein, Abu Bakr Soliman Mohammad, Mohamed Ibrahim et al.

Summary

The paper describes the NGU_CNLP team's entry to the WANLP 2022 shared task on detecting propaganda in Arabic tweets. Subtask 1 is multi-label classification: a tweet receives any of 17 propaganda techniques, or a "no technique" label, giving 18 labels. Subtask 2 requires locating the character spans where each technique appears. The training data was tiny and badly imbalanced, and a bag-of-words baseline scored zero on most classes.

The team made three main changes:

  • They pooled and re-split the provided sets with multi-label stratification.
  • They machine-translated the English SemEval-2020 Task 11 propaganda data into Arabic, mapping its labels onto the WANLP scheme.
  • They fine-tuned AraBERT as a 5-fold ensemble with averaged probabilities and a tuned decision threshold.

This system ranked first of 14 in subtask 1. For span detection, they converted spans to BIO tags and fine-tuned the Arabic Marefa-NER model, which placed third of three. The work shows that translated data from a related resource can partly make up for missing labels in a low-resource language.

Key points

  • Subtask 1 data: 504 train, 52 dev and 52 dev_test tweets. Tweets average 1.75 labels and carry up to 5. Eight of the 18 labels have fewer than 10 training examples.
  • Baseline: a Linear SVM over bag-of-words reached micro-F1 0.44, but most individual classes scored 0.
  • Augmentation: SemEval-2020 Task 11 sentences were translated with the RapidAPI Translation tool, adding 3,938 examples and growing the training pool from 504 to 4,442.
  • Subtask 1 model: AraBERT with max length 128 and learning rate 1e-4. A fraction of 0.7 of the extra data was sampled into training. The 5-fold ensemble used a 0.4 threshold and scored micro-F1 0.649 on 440 test tweets, ahead of the runner-up at 0.609.
  • Subtask 2: a Marefa-NER model (XLM-RoBERTa tokenizer, 24 layers, 16 heads) was fine-tuned on BIO tags at learning rate 1e-5. A span carrying two techniques occurred 53 times in train. The paper reports a third-of-three placement, but the extracted text gives no span score.

Interesting ideas

  • I1. Machine-translating a labelled corpus from a neighbouring language and mapping its label names onto the target scheme fills in rare classes cheaply. Sampling only part of that extra data limits the noise it adds.
  • I2. When the provided train, dev and test splits have mismatched label distributions, pooling them and re-splitting with multi-label stratification gives fairer development numbers.
  • I3. An ensemble of k-fold models with averaged probabilities and one empirically chosen cut-off is a simple, robust way to make multi-label decisions.
  • I4. Preprocessing was limited to operations that keep character order and offsets intact, so predicted spans still match the original text.
  • I5. When overlapping labels had to be collapsed into BIO, they dropped the most frequent technique and kept the rarer one, protecting minority classes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could borrow translated or remapped annotated corpora to boost sparse frame or speech-act classes, and tune how much borrowed silver data to mix in.
  • I2 (learn): Evaluation sets for rare constructions or frame types could be re-stratified so that negative controls and rare labels appear in every fold.
  • I3 (frames): Speech-act and irony detectors that can assign several labels at once could average fold models and set an explicit, inspectable threshold.
  • I4 (tokens): The tokenizer and normalizer should keep a reversible character-offset map, so every token and tag can be traced back to exact source positions.
  • I5 (frames): Where overlapping role or frame spans must be flattened, Mova could keep the rarer, more informative label and log the dropped one for explainability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New Giza UniversityGiza · Egypt3 works · 54 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Samir Hussein, Abu Bakr Soliman Mohammad, Mohamed Ibrahim, Laila Hesham Afify and Samhaa R. El-Beltagy. 2022. NGU CNLP atWANLP 2022 Shared Task: Propaganda Detection in Arabic. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.66

@inproceedings{hussein2022ngu,
  title = {{NGU CNLP atWANLP 2022 Shared Task: Propaganda Detection in Arabic}},
  author = {Ahmed Samir Hussein, Abu Bakr Soliman Mohammad, Mohamed Ibrahim, Laila Hesham Afify and Samhaa R. El-Beltagy},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.66},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.66}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bolsover Gillian and Philip Howard. Computational propaganda and political big data: Moving toward a more critical research agenda. In Big Data, pages 273–376, April 2017. *Figure 5: F 1 -score of our Model during training for Subtask 2* ↗
  2. Akshay Jain and Amey Kasbe. Fake news detection. In 2018 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pages 1–5, 2018. find
  3. Hani Al-Omari, Malak Abdullah, Ola Altiti, and Samira Shaikh. Justdeep at nlp4if 2019 task 1: Propaganda detection using ensemble deep learning models. Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, 2019. ↗
  4. Shaina Raza and Chen Ding. Fake news detection based on news content and social contexts: a transformer-based approach. International Journal of Data Science and Analytics, page 335–362, 2022. *Figure 6: Training Loss of our Model for Subtask 2* ↗
  5. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics, 10:178–206, 02 2022. After completing the whole training process, the ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and model with the highest F 1 -Score was retrieved and Kristina Toutanova. BERT: pre-training of deep bidi-adopted. The best model results are shown in Ta-rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, edible 2. Training 0.1637 Validation 1.2753 F 1 -score 0.1669 tors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. Accuracy 0.7815 *Table 2: Model’s validation results for subtask 2* find
  7. Miller C. R. The techniques of propaganda. from “how to detect and analyze propaganda,”. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, an address given at Town Hall. The Center for learning, 1939. find
  8. Jinfen Li, Zhihao Ye, and Lu Xiao. Detection of propaganda using logistic regression. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 119–124, Hong Kong, China, November 2019. Association for Computational Linguistics. find
  9. Shehel Yoosuf and Yin Yang. Fine-grained propaganda detection with fine-tuned BERT. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 87–91, Hong Kong, China, November 2019. Association for Computational Linguistics. find
  10. Tariq Alhindi, Jonas Pfeiffer, and Smaranda Muresan. Fine-tuned neural models for propaganda detection at the sentence and fragment levels. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 98–102, Hong Kong, China, November 2019. Association for Computational Linguistics. find
  11. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics.
  12. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online), December 2020. International Committee for Computational Linguistics. ↗
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May 2020. European Language Resource Association. ↗
  14. Zhiyong He, Zanbo Wang, Wei Wei, Shanshan Feng, Xianling Mao, and Sheng Jiang. A survey on recent advances in sequence labeling from deep learning models, 2020. ↗
  15. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hussein-2022-ngu-cnlp-atwanlp-2022-shared/