📄 iCompass at WANLP 2022 Shared Task: ARBERT and MARBERT for Multilabel Propaganda Classification of Arabic Tweets

Summary

The paper presents the iCompass entry to subtask 1 of the WANLP 2022 shared task: deciding which propaganda techniques, if any, appear in an Arabic tweet. Each tweet can carry several techniques, so this is multilabel classification. The authors merged the official train and dev files into a 556-tweet corpus. Only 18 of the label types actually occur, and their frequencies are extremely skewed. The tweets were cleaned by removing emojis, links, special characters and stop words, and by normalizing the text. Two Arabic BERT-style encoders were then fine-tuned: ARBERT, pretrained on Modern Standard Arabic, and MARBERT, pretrained on dialect-heavy tweets. The authors compared runs with and without an extra pooling head and with and without 5-fold cross-validation, repeating each setup several times. On development data the best configurations scored about 0.56–0.57 micro F1. The submitted MARBERT system scored 0.597 micro F1 on the test set and placed fifth. Macro F1 stayed low, which shows how hard rare techniques are to learn from such limited data.

Key points

• Data: 556 tweets from task1_train.json plus task1_dev.json, with 18 label types present. Loaded Language appears in 346 tweets, while Presenting Irrelevant Data (Red Herring) appears in only 2.
• Classification head: the token states go through two 1D global pooling layers, one averaging and one taking the maximum, and their outputs are concatenated before dropout and an output layer. Training ran up to 10 epochs with early stopping, keeping the checkpoint with the lowest validation loss.
• Evaluation protocol: each configuration was run at least 5 times and reported as mean ± std micro F1.
  • ARBERT, no pooling, with CV: 0.567 ± 0.028.
  • MARBERT, pooling, with CV: 0.562 ± 0.012.
  • Macro F1 for these two: 0.243 (ARBERT) and 0.282 (MARBERT).
• Submitted system: MARBERT with pooling and 5-fold CV scored 0.597 micro F1 and 0.191 macro F1 on the gold test set, ranking 5th.
• Stated limitations: the authors attribute the weak results to the small corpus and scarce classes such as Whataboutism. They plan to try augmentation and resampling.

Interesting ideas

• I1. Report variance across repeated runs: each setup was rerun several times and reported as mean with standard deviation, and the submitted MARBERT configuration had a slightly lower mean but a clearly smaller spread than the top ARBERT one. This keeps small datasets from crowning a lucky seed.
• I2. Show micro and macro F1 together: micro F1 near 0.6 next to macro F1 below 0.3 makes it obvious that frequent labels hide failure on rare ones.
• I3. Combine mean and max pooling in the head: concatenating average and max pooling captures both the overall tone and the single strongest signal in the token states. That suits techniques triggered by one charged word, such as Loaded Language or Name calling.
• I4. Pick the encoder to match the text register: the paper compares an encoder pretrained on standard written Arabic with one pretrained on tweets. Choosing the pretraining domain to match the target text is a useful explicit experimental axis.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation harness could rerun each training configuration across seeds and record the spread, treating gains smaller than that spread as noise.
• I2 (frames): when scoring speech-act or semantic-role labels, Mova should print both averages so rare roles that are never predicted cannot hide behind common ones.
• I3 (frames): an auxiliary irony or rhetorical-device detector in prag could combine both pooled views over clause tokens, while keeping the max-pooled token as an explainable pointer to the trigger word.
• I4 (tree): gparse could be benchmarked with encoders pretrained on edited prose versus informal web text before parsing social-media input, with UD LAS compared per register.