📄 AraBERT Model for Propaganda Detection

Summary

The paper presents SIREN AI's entry to the WANLP 2022 shared task on detecting propaganda techniques in Arabic tweets. The task is multi-label: a tweet may use several persuasion techniques, such as Loaded language or Name calling, or none. The team fine-tuned the second version of AraBERT on a very small, skewed training set. The model outputs one probability for each of the 18 labels, and a hand-tuned decoding step turns those scores into label sets. That step keeps at most five candidates, drops low scores below a tuned cutoff, and uses the no-technique label as a reference point. The submitted system reached a Micro-F1 of 0.4108 on the test table. The authors also report 0.578 on the evaluation data and an unsubmitted 0.61 with augmentation and reweighting of the classifier layer. Training-set Micro-F1 was about 0.88, so the authors conclude the model badly overfits because the data is too small and unrepresentative. Their proposed remedies are regularization, cheap text augmentation, class weighting and more annotation. The work matters mainly as a baseline showing how hard low-resource Arabic propaganda detection remains.

Key points

• The data came from the shared-task organizers: 504 training tweets, plus 52 development and 52 test tweets with the same label distribution.
  • There were 18 labels, including no-technique, with at most five labels per tweet and usually one or two.
• The model was AraBERT v2 in a BERT-base configuration (12 encoder blocks, 12 heads, 768 hidden size, 512 max length), pretrained on about 70M sentences.
  • Text was cleaned with the AraBERT preprocessor, which removes diacritics, tatweel and HTML and normalizes digits.
• Training settings:
  • Adam with learning rate 2e-5 and epsilon 1e-8.
  • 25 epochs, batch size 16 with gradient accumulation of 2.
  • About five minutes on a V100 GPU in Colab.
• Decoding used the top-5 probabilities, a threshold of 0.35 and a special rule built around the no-technique label.
  • The paper's wording of this rule is somewhat unclear.
• Reported scores:
  • Test: Micro-F1 0.4108, Macro-F1 0.108, training loss 0.19, validation loss 0.3.
  • Evaluation set: Micro-F1 0.578.
  • Unsubmitted variant with augmentation: 0.61.
  • Training set: about 0.88, a clear sign of overfitting.

Interesting ideas

• I1. Choose the decision threshold so that the predicted label distribution resembles the gold one, rather than tuning only for a score; on tiny dev sets this is a more stable signal.
• I2. Cap the number of predicted labels at the maximum count ever seen in annotation, so the data itself sets a structural limit on output size.
• I3. Treat the "none" label as an internal reference point: real labels must outscore it to be kept, otherwise the system falls back to none.
• I4. Report training-set and held-out scores side by side, so a large gap (0.88 vs ~0.4–0.58 here) diagnoses overfitting rather than model weakness.
• I5. Fix the configuration on separate splits first, then retrain on all labeled data merged together for the final model; this squeezes the most out of very small corpora.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver-data filters could pick confidence cutoffs so that the accepted analyses match the label frequencies seen in gold treebank data, which also works as a sanity check on calibration.
• I2 (frames): When assigning VerbNet roles or speech-act tags, limit the candidates per frame or utterance to the attested maximum, which prunes implausible over-labelled analyses.
• I3 (answer): A reader could compare each candidate answer against an explicit "cannot tell from the text" option and abstain when nothing beats it, giving an explainable refusal.
• I4 (learn): Every trained component, such as the perceptron parser or repair rules, could log this gap by default next to negative controls, flagging when gains come from memorization.
• I5 (tree): For small or newly converted UD treebanks, tune gparse settings on held-out splits, then retrain on the union before release.