📄 LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT's pretrain-finetune discrepancy for Arabic offensive language identification

Summary

This paper presents the winning system for the Arabic track of OffensEval 2020 (SemEval-2020 Task 12, sub-task A). The task is a binary decision: is a tweet offensive (OFF) or not (NOT)? The authors fine-tune AraBERT, a BERT BASE-sized model pretrained on Arabic text. Their main contribution is how they treat emojis.

In the tweet itself, each emoji becomes the special token [MASK]. The emoji's English description is translated into Arabic and placed in a second input segment, separated by [SEP]. The motivation is that [MASK] appears during pretraining but almost never in task data. Keeping it in the input during fine-tuning and inference narrows this pretrain-finetune mismatch.

Text is segmented with Farasa before AraBERT tokenization. Token embeddings are max-pooled into a single tweet vector, which a sigmoid output layer classifies.

The authors compared three preprocessing variants. The [MASK]-plus-side-segment version was the strongest, and it reached 90.17% macro F1 on the hidden test set, ranking first. The work shows that cheap, pretraining-aware input design can measurably improve a fine-tuned transformer.

Key points

• Dataset: the Arabic data comes from Mubarak et al. (2020) and contains 10,000 labelled tweets.
  • Split: 7,000 train, 1,000 dev, 2,000 test.
  • NOT tweets outnumber OFF tweets roughly two to one.
• Model:
  • AraBERT with the BERT BASE configuration: 12 layers, 768 hidden units, 12 heads, about 110M parameters.
  • Pretrained on about 70M Arabic sentences (~24GB of text).
  • Fine-tuned with max sequence length 128, 5 epochs, batch size 32, Adam at 1e-4 with warmup and weight decay.
  • Built with TensorFlow 2.0 and keras-bert.
• Variants compared (Table 1):
  • Vanilla, with no emoji handling: 88.82% macro F1.
  • AraBERTEmojisIN, with emojis replaced inline by their Arabic meanings: 88.42%.
  • AraBERTEmojisOUT, with [MASK] inline plus a separate meanings segment: 89.06% macro F1, 93.9% accuracy, and the best AUC.
• Official result: AraBERTEmojisOUT scored 90.17% macro F1 on test, ranking first.
  • Per class, NOT reached about 96% precision and recall.
  • OFF reached 83% precision and 85% recall; the authors attribute the gap to class imbalance.
• Representation: global max pooling over all token vectors, not the [CLS] vector, feeding a sigmoid trained with binary cross-entropy.

Interesting ideas

• I1. Instead of feeding a pretrained encoder symbols it has rarely seen, swap them for a token it learned during pretraining ([MASK]), so fine-tuning inputs look more like pretraining inputs.
• I2. Keep the main sentence clean by moving extra information (here, emoji glosses) into a separate input segment rather than splicing it inline; inline insertion actually scored slightly worse than doing nothing.
• I3. Turning an emoji into a natural-language description in the working language makes its emotional or evaluative content readable by an ordinary text model.
• I4. The evaluation holds the model fixed and varies only preprocessing across three variants, so any gain can be credited to the input design itself.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could mask emojis, URLs or odd symbols with a known placeholder before encoding, so the transformer parser is not confused by unfamiliar subword pieces.
• I2 (tokens): the tokenizer could emit a single placeholder token for an emoji or symbol in the sentence stream and attach its gloss as side metadata, so the UD tree is not distorted by injected words.
• I3 (frames): prag could read emoji glosses as explicit cues for speech act, sentiment or irony detection, keeping the cue traceable rather than hidden in embeddings.
• I4 (learn): when testing a tokenizer or normalization change, train otherwise identical models that differ in that one step, and keep the "no change" variant as a built-in control.