📄 IITD at WANLP 2022 Shared Task: Multilingual Multi-Granularity Network for Propaganda Detection

Summary

The paper describes a second-place system for the WANLP 2022 shared task on detecting propaganda in Arabic tweets. The task had two parts: deciding which of 20 propaganda techniques a tweet uses, and marking the exact text span of each technique. The data is a small set of Arabic tweets, called AraTweet by the authors. For the first part, XLM-R is fine-tuned with one sigmoid output per technique and a 0.5 threshold. For the span part, the multi-granularity network (MGN) of Da San Martino et al. is used with mBERT in place of English BERT, giving mMGN. The authors also tested whether the much larger English Propaganda Techniques Corpus (PTC) could help, both directly and after machine translation into Arabic with labels projected by awesome-align. It did not. Pretraining on English PTC and then continuing on Arabic lowered F1 substantially, and the translated data gave no gain. The authors attribute this to the large domain gap between news articles and tweets. This is a useful negative result about cross-lingual transfer for span-level annotation.

Key points

• AraTweet has 504 train, 103 dev (dev + dev_test combined) and 323 test tweets, covering 20 techniques. Loaded language is by far the most common technique (446 train instances), several techniques are almost absent, and 95 training tweets contain no propaganda.
• Subtask 1 micro-F1 by encoder: XLM-R 60.9, AraBERT 59.4, mBERT 54.3. Macro-F1 was very low for all three (8.1–18.7), which shows how poorly rare classes are handled.
• Subtask 2: mMGN trained only on AraTweet reaches 35.5 F1 on the test set. The metric is a micro-F1 variant that gives credit for partial span overlap.
• The system placed second in both subtasks, out of 14 and 3 teams respectively.
• Cross-lingual results:
  • Training on English PTC and then continuing on AraTweet cost 10.2 F1 points.
  • Translated PTC (TransPTC) scored 0.6 F1 above raw PTC in zero-shot use.
  • Continued training from a propaganda-only translated subset (TransPTC+) left results unchanged.

Interesting ideas

• I1. MGN adds a learnable sentence-level gate that pushes down token-level label probabilities when the whole sentence looks clean, so a coarse judgement constrains fine-grained tagging.
• I2. Multi-label prediction with independent sigmoids and a fixed threshold naturally allows an empty label set as a valid output, with no extra "none" class.
• I3. Creating annotated data for another language by machine translation plus word-alignment projection of span labels is cheap, but here it gave no gain because of domain mismatch. Borrowed silver data should therefore be checked against the target domain.
• I4. Reporting macro-F1 next to micro-F1 exposed that strong-looking scores hid near-total failure on rare classes.
• I5. Projecting span annotations through word alignment carries structure across a translation instead of re-annotating from scratch.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): an utterance-level irony or speech-act decision could gate token-level cue marking, so irony cues are only reported when the sentence as a whole is judged ironic.
• I2 (frames): implicature or speech-act labels could be scored independently per clause, so a clause can have several labels or none without forcing a single choice.
• I3 (learn): before adding projected or borrowed silver trees or frames, run a held-out check against a native-domain baseline to confirm the data does not silently hurt.
• I4 (learn): evaluation reports should always include a per-class or macro view, so that rare frames or rare dependency relations cannot hide behind frequent ones.
• I5 (back): when translating through the meta-world, entity and frame spans could be carried onto the output text via alignment, keeping each generated span explainable.