📄 ALT Submission for OSACT Shared Task on Offensive Language Detection

Summary

The ALT team from Qatar Computing Research Institute entered the OSACT4 shared task on Arabic tweets. The task had two parts: flagging offensive language (Subtask A) and flagging hate speech (Subtask B). The organizers supplied 10,000 labelled tweets with a strong class imbalance: about 19% are offensive and only about 5% are hate speech. Ranking used macro-averaged F1. After a fixed four-step cleanup of the tweets, the authors compared many classifiers:

• SVMs on tf-idf character and word n-grams, plus Mazajak Twitter embeddings
• FastText
• a feed-forward network
• CNN and CNN-BiLSTM
• fine-tuned M-BERT

Tuned SVMs beat every neural model when used alone. Because the model families differ, the authors expected their errors to differ too, so they combined them. For Subtask A, a majority vote over two SVMs, CNN-BiLSTM and M-BERT took 1st place with a test F1 of 90.51%. For Subtask B, three SVMs labelled a tweet as hate speech if any single one said so, which placed 4th with an F1 of 80.63%. The paper shows that cheap, diverse ensembles work well for low-resource, noisy, skewed classification.

Key points

• Data: 10,000 Arabic tweets, split 7,000 train / 1,000 dev / 2,000 test. All hate speech counts as offensive by definition.
• Preprocessing: the authors applied four steps in order:
  1. drop words containing non-Arabic characters
  2. strip diacritics
  3. remove punctuation
  4. collapse repeated letters into one
• Best single model on dev for Subtask A: an SVM on Mazajak SG-250 embeddings plus char [1-5] and word [1-3] grams, with F1 90.4. By comparison, CNN-BiLSTM reached 87.8, M-BERT 83.3 and FastText 79.4.
• Domain mismatch: the authors attribute M-BERT's weaker score to its formal Wikipedia pretraining, which differs from informal Twitter text.
• Error analysis for Subtask A: misclassified tweets fell into three groups: missing context, out-of-vocabulary words, and ambiguous words (for example, a word read as offensive in a harmless sentence).
• Hate speech remains hard: the dev confusion matrix for Subtask B shows 16 of 42 hate-speech tweets missed. Data augmentation is planned as future work.

Interesting ideas

• I1. Pick ensemble members for how different their errors are, not only for top scores; mixing SVMs with neural models beat every single system.
• I2. For a very rare class, switch from majority voting to an "any member says yes" rule to trade precision for recall; this won on dev for hate speech.
• I3. A character-level CNN learned jointly with word embeddings covers words missing from the pretrained vocabulary and captures morphology.
• I4. Domain fit of pretraining mattered more than model power: Twitter-trained Mazajak embeddings beat Wikipedia-trained M-BERT.
• I5. Sorting a sample of errors into named causes (context, out-of-vocabulary, ambiguity) turns failures into a research agenda.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when agreeing silver data across witness parsers, prefer witnesses built on different principles (perceptron vs gparse), so that agreement is more informative.
• I2 (frames): rare pragmatic labels such as irony or implicature could use a union vote among detectors as a high-recall candidate stage, followed by an explainable confirming check.
• I3 (tokens): character-based fallback features could help guess lemma and POS for unseen or misspelled informal English words, before the tagger gives up.
• I4 (tree): when training gparse for fables, school texts or forum text, weigh treebank and pretraining domain match explicitly, and report scores per domain.
• I5 (learn): tag evaluation failures with a fixed cause taxonomy, so that self-induced repair rules can target one cause class at a time and be checked against negative controls.