Elamur Research⌕
☰
home / papers / 2022 / CNLP-NITS-PP at WANLP 2022 Shared Task: Propaganda Detection in Arabic using Data Augmentation and AraBERT Pre-trained Model

CNLP-NITS-PP at WANLP 2022 Shared Task: Propaganda Detection in Arabic using Data Augmentation and AraBERT Pre-trained Model

Paper · 2022 · Sahinur Rahman Laskar, Rahul Singh, Abdullah Faiz Ur Rahman Khilji et al.

Summary

The paper describes the CNLP-NITS-PP team's entry to subtask 1 of the WANLP 2022 shared task, which focused on detecting propaganda in Arabic text. In this subtask, each tweet must be tagged with every persuasion technique it uses, drawn from an inventory of 21 techniques plus a "no technique" label. The main difficulty is the small amount of training data: only 504 annotated tweets. The authors note that social media users often write in bare or stemmed word forms. They therefore ran the Arabic Light Stemmer (Tashaphyne) over each training tweet and added both a stem version and a root version with the original labels. This tripled the training set to 1512 examples. They then fine-tuned the AraBERT twitter-base model, adapting a single-label example script to multi-label output. At inference, every label scoring at least 0.32 was kept. The system scored a micro-F1 of 0.602 and shared third place among 15 teams. Its macro-F1, however, was very low. The work shows that cheap morphological augmentation plus a domain-matched pretrained model can be competitive under severe data scarcity.

Key points

  • Data splits: 504 train, 104 validation and 440 test tweets. Augmentation added 1,008 synthetic examples, giving 1512 training samples in total.
  • Model: AraBERT twitter-base (bert-base-arabertv02-twitter) with the default ArabertPreprocessor. Training used dropout 0.1, Adam, batch size 16 and 5 epochs. It ran on a single Quadro P2000 GPU and took under 5 minutes.
  • Labels were one-hot encoded for multi-label training, and the F1 scorer was modified to handle several labels per tweet.
  • Decision threshold: on validation, correct labels mostly scored above 0.30, with a clear gap below that point. The cut-off was set at 0.32.
  • Result: micro-F1 0.602, tied for 3rd of 15 teams; the winner scored 0.649. The team's macro-F1 was only 0.068, against 0.185 for the top team.

Interesting ideas

  • I1. Cheap augmentation by morphological reduction: adding stem-only and root-only copies of each training text with unchanged labels makes a classifier tolerant of ungrammatical or base-form input without any new annotation.
  • I2. Pick the decision threshold from the shape of validation scores: the authors searched for a natural gap separating correct from incorrect labels instead of defaulting to 0.5. This works whenever scores cluster on two sides of a gap.
  • I3. Always report micro- and macro-F1 together: here 0.602 micro sits beside 0.068 macro, showing the system mostly succeeded on frequent labels and largely missed rare ones.
  • I4. Persuasion techniques as a fine-grained, multi-label pragmatic inventory: labels like Straw Man, Whataboutism, Loaded Language and Doubt describe what a text is doing to its reader, not what it says.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Lemma-substituted copies of training sentences could harden the tagger and lemmatizer against sloppy informal English, such as missing inflections in chat-style text.
  • I2 (learn): When accepting silver annotations from witness parsers, the confidence cut-off could be placed at an observed gap in the agreement-score distribution rather than at an arbitrary value.
  • I3 (learn): Evaluations of rare categories, such as rare semantic roles or speech acts, should report per-class or macro scores so that collapse onto common classes is caught early.
  • I4 (frames): The prag crate could add such techniques as speech-act-like annotations alongside irony and implicature, each one explained by the frames that trigger it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Institute Of Technology SilcharSilchar · India6 works · 72 citations

Also on the byline: Adamas University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sahinur Rahman Laskar, Rahul Singh, Abdullah Faiz Ur Rahman Khilji, Riyanka Manna, Partha Pakray and Sivaji Bandyopadhyay. 2022. CNLP-NITS-PP at WANLP 2022 Shared Task: Propaganda Detection in Arabic using Data Augmentation and AraBERT Pre-trained Model. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.65

@inproceedings{laskar2022cnlp,
  title = {{CNLP-NITS-PP at WANLP 2022 Shared Task: Propaganda Detection in Arabic using Data Augmentation and AraBERT Pre-trained Model}},
  author = {Sahinur Rahman Laskar, Rahul Singh, Abdullah Faiz Ur Rahman Khilji, Riyanka Manna, Partha Pakray and Sivaji Bandyopadhyay},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.65},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.65}
}

References

10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics.
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021a. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021b. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. find
  6. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. Detecting propaganda techniques in memes. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6603–6617, Online. Association for Computational Linguistics. ↗
  7. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021b. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. ↗
  8. Vorakit Vorakitphan, Elena Cabrio, and Serena Villata. 2021. PROTECT - A pipeline for propaganda detection and classification. In Proceedings of the Eighth Italian Conference on Computational Linguistics, CLiC-it 2021, Milan, Italy, January 26-28, 2022, volume 3033 of CEUR Workshop Proceedings. CEUR-WS.org. find
  9. Seunghak Yu, Giovanni Da San Martino, Mitra Mohtarami, James Glass, and Preslav Nakov. 2021. Interpretable propaganda detection in news articles. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1597–1605, Held Online. IN-COMA Ltd. ↗
  10. Taha Zerrouki. 2012. Tashaphyne, arabic light stemmer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/laskar-2022-cnlp-nits-pp-wanlp-2022/