📄 iCompass at Arabic Hate Speech 2022: Detect Hate Speech Using QRNN and Transformers

Summary

This system paper describes the iCompass entry to the OSACT 2022 shared task on fine-grained hate speech detection in Arabic tweets. The task has three subtasks: offensive vs. not (A), hate speech vs. not (B), and the type of hate (C), over six categories such as religion, gender and social class. The authors compared mT5, AraBERT, ARBERT and MARBERT on the development data. MARBERT, which was pre-trained on dialectal tweets, performed best, so it became the shared encoder of a hard-parameter-sharing multi-task network. Each subtask gets its own head: a convolution layer, then a bidirectional QRNN, then dense layers. To ease the scarcity and skew of the labels, they relabelled tweets from the Levantine L-HSAB corpus to fit the task scheme and resampled the classes. On the test set the system reached macro-F1 of 0.839 (A), 0.831 (B) and 0.528 (C). These scores are far above the organisers' baselines. The fine-grained subtask stays weak because some hate classes have almost no examples.

Key points

• The task data has about 8.9k training tweets in the released split. Around 35% are offensive and 11% are hate speech, and the disability class has zero training instances.
• External data came from L-HSAB (468 hate, 1728 offensive, 3650 normal tweets). These tweets were manually relabelled, after which minority classes were over-sampled and majority classes under-sampled.
• Preprocessing was deliberately light:
  • non-Arabic tokens were dropped, including USER, URL and emojis;
  • hashtags were split into words;
  • extra whitespace was removed.
• Each subtask head has the same core: Conv1D (128 units, kernel 3), then a BiQRNN (256 units), then a 64-unit dense layer.
  • Subtask A ends in a sigmoid output with a 0.75 threshold.
  • Subtasks B and C end in a 7-way softmax.
• Training ran for 16 epochs in two phases:
  • first 6 epochs: MARBERT frozen, learning rate 1e-3, Adam, batch size 350;
  • last 10 epochs: MARBERT unfrozen, learning rate 1e-4.
• Development baselines had macro-F1 of 0.394, 0.472 and 0.135 for the three subtasks.

Interesting ideas

• I1. Train the task heads first with the pretrained encoder frozen and a high learning rate, then unfreeze everything at a lower rate; this stops random heads from corrupting the encoder early on.
• I2. Add an extra output whose only job is to penalise one specific confusion between neighbouring labels (offensive read as hate); this alone lifted the fine-grained subtask by about 0.04.
• I3. Enlarge a scarce dataset by relabelling a related corpus from a neighbouring dialect into the target label scheme, rather than using its original labels.
• I4. Report scores with and without the external resource side by side, so the contribution of added data is visible per subtask rather than assumed.
• I5. Split hashtags into their component words instead of deleting them, so the content they carry stays readable to the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could warm up its arc and label scorers on a frozen transformer before joint fine-tuning, which should give more stable runs on small treebanks.
• I2 (tree): a similar auxiliary penalty could target known confusable deprel pairs such as obj/iobj or nmod/obl, where most labelled-attachment errors cluster.
• I3 (learn): treebanks in other UD dialects or annotation styles could be converted and checked by witness parsers before being added to silver training data.
• I4 (learn): every silver-data batch could be evaluated as an ablation against a gold-only control, which acts as a built-in negative control.
• I5 (tokens): the en tokenizer could segment hashtags and camelCase tokens into words before tagging, giving the lemmatizer and parser real words instead of opaque blobs.