📄 Offensive language detection in Arabic using ULMFiT

Summary

The paper describes an entry for the OSACT4 shared task (OffenseEval 2020) on Arabic tweets. The task has two binary labels: whether a tweet is offensive, and whether it is hate speech. Both label sets are heavily skewed toward the harmless class. The authors start from a ULMFiT language model that Khooli (2019) had already pre-trained on Arabic Wikipedia. They adapt it to the tweet text and then train classifiers on top using fastai with an AWD-LSTM backbone. Following the original ULMFiT recipe, they build separate left-to-right and right-to-left models and average their predictions. Preprocessing is deliberately generic: whitespace splitting, a frequency cutoff, and fastai special tokens.

On the development data, combining the two directions gave the best macro F1 in both subtasks. On the test set, offensive detection reached a macro F1 of 77. Hate detection lagged at 58, largely because the system recalled few minority-class tweets. The work shows that ULMFiT-style transfer is a workable, low-effort baseline for Arabic. It also shows that accuracy and weighted scores hide poor performance on rare classes.

Key points

• Data: 10k Arabic tweets from OSACT4, split 7k train / 1k dev / 2k test, with each tweet labelled for both subtasks.
  • Subtask A (offensive): roughly 1.4k positive tweets in train, 179 in dev.
  • Subtask B (hateful): roughly 0.4k positive tweets in train, only 44 in dev.
• Model and setup:
  • Backbone: a 3-layer AWD-LSTM (Merity et al.) trained with fastai.
  • Starting point: Khooli's Arabic Wikipedia ULMFiT model.
  • LM fine-tuning: done on all available unlabelled task text, i.e. train plus dev.
  • Training schedule: gradual unfreezing.
  • Hardware: a Titan X GPU with batch size 64.
• Dev macro F1, subtask A: 74 (forward), 78 (backward) and 80 (averaged).
• Dev macro F1, subtask B: 60, 61 and 61, even though accuracy sits around 96.
• Test results:
  • Subtask A: accuracy 86, macro F1 77.
  • Subtask B: accuracy 95, macro F1 58.
• Future work proposed: morphology-aware Arabic tokenization, and using a language model to generate extra minority-class examples, alone or alongside random oversampling or SMOTE.

Interesting ideas

• I1. Training the same architecture once reading left-to-right and once right-to-left, then averaging outputs, gave a nearly free improvement on the rare class; direction reversal is a cheap source of model diversity.
• I2. The paper shows how a 95–96% accuracy can coexist with a macro F1 near 60, so weighted or accuracy-style scores are misleading whenever the rare label is the one that matters.
• I3. Before training the classifier, the language model is first adapted to the raw text of the target domain, using unlabelled material that labelled training would otherwise waste.
• I4. Instead of discarding surface cues, the tokenizer turns repetition and capitalization into explicit marker tokens, so the model still sees emphasis without bloating the vocabulary.
• I5. The authors suggest using a generative language model to synthesize additional examples of the underrepresented class, as an alternative or complement to classic oversampling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): a right-to-left variant of the perceptron parser or gparse could serve as an extra, partially independent witness when voting on silver trees.
• I2 (learn): evaluation reports should give macro and per-label scores for rare dependency relations, rare frames, and irony, so gains on frequent cases cannot mask failures on scarce ones.
• I3 (tokens): the tagger and lemma models could be refreshed on unlabelled fables, school texts, and word problems before supervised training, narrowing the gap between treebank genre and target genre.
• I4 (frames): keeping such emphasis markers as token features would give the irony and speech-act components an explainable signal (shouting, letter stretching) rather than losing it in normalization.
• I5 (learn): an LLM teacher tier could generate extra sentences for rare constructions or rare frames, which witness parsers would then vet before the sentences enter training.