📄 aiXplain at Arabic Hate Speech 2022: An Ensemble Based Approach to Detecting Offensive Tweets

Summary

The paper describes aiXplain's entry for subtask A of the OSACT5 shared task, which asks whether an Arabic tweet is offensive. The data is a 12,698-tweet corpus labelled by three annotators. It is split into train, dev and test sets, and offensive tweets are the minority class. The team first balanced the training data in two ways: pseudo-labelling scraped tweets with a fine-tuned AraBERT model, and contextual word substitution. Preprocessing stripped noise from the text. Emojis were handled separately: they were translated into Arabic, and some were given manually chosen regional meanings. Several classifiers then fed a meta-learner (XGBoost, auto-sklearn or a linear layer), with emoji-based scores added as extra features. AraBERT alone dominated the other models, and the full ensemble added only a small gain. The best test run reached about 0.85 macro F1, far above the majority baseline. The authors see dialect variation as the main obstacle, since most Arabic tools assume MSA. They point to emojis as a largely language-independent cue worth exploring further.

Key points

• Data split: 8,887 train, 1,270 dev and 2,541 test tweets. Subtask A is roughly 65% not-offensive and 35% offensive.

• Augmentation used two methods:

  • Confident pseudo-labels from AraBERTv0.2-Twitter-base on external tweets.
  • NLPAug contextual substitution of about 30% of the words, which doubled the offensive class.

• Ensemble members were:

  • AraBERTv0.2-Twitter-large (Adam, lr 1e-5).
  • Mazajak 250M CBOW embeddings, average-pooled.
  • Char-trigram plus word-bigram TF-IDF features.
  • MUSE sentence vectors.
  • An emoji score built from how often each emoji appears in offensive versus clean tweets.

• Dev macro F1 by model:

  Model	Dev macro F1
  Full ensemble	0.85
  AraBERT	0.84
  Mazajak	0.64
  Emoji score	0.51

• Test results:

  Run	Accuracy	Macro F1
  Auto-sklearn meta-learner	0.864	0.849
  Linear meta-learner	0.858	0.843
  Majority baseline	—	0.394

Interesting ideas

• I1. The ablation table shows that stacking weaker feature extractors on a strong transformer barely moved the score, so the extra members carried little independent signal.
• I2. They expanded the minority class by labelling outside data with a fine-tuned model and keeping only high-confidence predictions.
• I3. Emojis were turned into words in the target language, with hand-picked readings for regionally specific ones, so models without emoji knowledge could still use them.
• I4. Each emoji got a simple, inspectable polarity score from its co-occurrence with offensive labels, and these scores served as a high-level feature.
• I5. Substitutions were capped at a few words per sentence because heavier changes drifted from the original meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before adding another witness parser to silver-data agreement, measure how much it actually changes the agreed output beyond the strongest witness, and drop witnesses that only echo it.
• I2 (learn): Mova's silver data could keep only sentences where the teacher is confident, applied specifically to rare constructions that treebanks underrepresent.
• I3 (tokens): The en tokenizer could map emojis and similar symbols to a gloss lemma with a POS-like tag, so later stages see a meaningful token instead of an unknown one.
• I4 (frames): The prag crate could keep a transparent cue table of this kind for symbols and markers, as explainable evidence in irony or hostile speech-act detection.
• I5 (tree): Treebank sentences could be augmented by swapping two or three words for contextually similar ones of the same POS while keeping the gold UD tree, which yields extra training data for the perceptron and gparse.