📄 CUET_sstm at ArAIEval Shared Task: Unimodal (Text) Propagandistic Technique Detection Using Transformer-Based Model

Summary

The paper describes the winning entry in Task 1 of the ArAIEval shared task at ArabicNLP 2024. The task is to find propaganda in Arabic news paragraphs and tweets and to mark the exact text spans, using a set of 23 technique labels. The training data are highly imbalanced: about three quarters of the texts contain no technique. To reduce this, the authors expanded the ten rarest classes with WordNet-based synonym replacement from nlpaug and shifted span offsets so the labels stayed valid. They normalized Arabic orthography, split texts word by word, and converted the spans to BIO tags, which turns the problem into a token-classification task with 47 classes. They compared classical classifiers (RF, SVM, MNB), a BiLSTM-CRF and three transformers: bert-base-arabic, Marefa-NER and AraBERT. A fine-tuned bert-base-arabic reached a micro-F1 of about 0.30 and placed first. The low absolute score shows how hard span-level propaganda detection remains. The authors name two main sources of error: overlapping spans that flat BIO tagging cannot represent, and the class imbalance that remains after augmentation.

Key points

• Data come from the ArAIEval Task 1 Arabic propaganda dataset. The splits are 6997 train (995 tweets, 6002 paragraphs), 921 dev and 1046 test samples. Roughly 74.1% of texts contain no technique.
• Augmenting the 10 least frequent techniques with nlpaug (WordNet source) increased the training set from 6997 to 9812 entries.
• Preprocessing:
  • removes emojis, URLs, HTML, numbers and punctuation;
  • strips Tashkeel;
  • unifies alif, taa marbuta and kaf variants.
• Word-by-word splitting aligned with the gold spans better than the marefa-ner or bert-base-arabic tokenizers.
• Test micro-F1 by model:
  • bert-base-arabic: ≈0.299 (P 0.314, R 0.286);
  • Marefa-NER: 0.281;
  • AraBERT: 0.279;
  • BiLSTM-CRF: 0.211;
  • SVM: 0.16;
  • MNB baseline: 0.112.
• bert-base-arabic was trained with learning rate 1e-6, weight decay 0.001, batch size 8 and up to 15 epochs, keeping the checkpoint with the lowest evaluation loss.

Interesting ideas

• I1. When subword tokenizers misalign with character-offset annotations, splitting on whole words and tagging every word that touches a span gives a simple and more consistent mapping.
• I2. The authors' main error source was overlapping labeled spans, which a single BIO layer cannot express, so the tag scheme must be chosen to match the structure of the annotation.
• I3. Synonym-replacement augmentation needs span indices recomputed after each substitution. The paper also found that augmenting rare classes unintentionally inflated a dominant one, Loaded Language.
• I4. To test whether small gaps between models were meaningful, the authors split the test set into four random chunks and compared scores on each. The best model led on three of the four.
• I5. The paper treats propaganda techniques as fine-grained span labels attached to wording. This makes manipulative rhetoric something that can be located and explained, not just a whole-document classification.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): This could serve as a fallback for projecting character-offset annotations from external corpora onto Mova's tokens, with the remaining misaligned cases logged instead of silently dropped.
• I2 (frames): Pragmatic labels such as irony, implicature or loaded wording can overlap, so prag should store them as separate span layers over the UD tree rather than as one flat tag sequence.
• I3 (learn): Any silver-data augmentation should re-check label anchoring and recompute the class distribution afterwards, treating unexpected growth of majority classes as a failure signal.
• I4 (learn): This cheap per-chunk ranking check can be added to parser and reader evaluations to flag improvements that only hold on part of the data.
• I5 (frames): prag could attach technique-like rhetorical tags (for example loaded language) to frame arguments, so a reader can say which words carry the persuasive force and why.