📄 iCompass at SemEval-2020 Task 12: From a Syntax-ignorant N-gram Embeddings Model to a Deep Bidirectional Language Model

Summary

The paper describes team iCompass's entry to SemEval-2020 Task 12, subtask A for Arabic. The task is a binary decision: is a social-media post offensive (OFF) or not (NOT)? The official data is mostly Egyptian dialect, with some other dialects mixed in. The authors expanded training with two dialectal abuse/hate corpora, L-HSAB (Levantine) and T-HSAB (Tunisian). They cleaned the text with regular expressions, leaving only Arabic script.

Three approaches were compared:

• Tw-StAR, a shallow syntax-ignorant n-gram embeddings model, tested alone and combined with an offensive-word lexicon;
• fine-tuned multilingual cased BERT-base;
• BERT's embedding matrix feeding a BiLSTM classifier.

On the development set, BERT+BiLSTM performed best, with accuracy 0.898 and macro F1 0.825. Tw-StAR had similar accuracy but a macro F1 of only about 0.33. Adding the extra dialect corpora raised macro F1 from 0.784 to 0.825. The work shows that contextual representations and cross-dialect data help for noisy, imbalanced, dialectal Arabic. It also shows that a word lexicon is fragile against negation.

Key points

• Official splits: TRAIN 7,000, DEV 1,000 and TEST 2,000 comments. With L-HSAB (5,846) and T-HSAB (6,039), the combined training pool was 18,885 items, of which 5,734 were OFF.

• Annotation reliability of the added corpora, measured by Krippendorff's alpha, was 76.5% for L-HSAB and 75% for T-HSAB. Their original three labels (Normal/Abusive/Hate) were used for the binary task.

• Dev results (accuracy / macro F1):

  Model	Accuracy	Macro F1
  Tw-StAR	0.813	0.331
  Tw-StAR+Lexicon	0.818	0.332
  BERT	0.821	0.791
  BERT+BiLSTM	0.898	0.825

• Data ablation for BERT+BiLSTM (macro F1):

  Training data	Macro F1
  SemEval only	0.784
  SemEval + L-HSAB	0.794
  SemEval + L-HSAB + T-HSAB	0.825

• Settings: BERT used batch 8, sequence length 128 and 3 epochs. BERT+BiLSTM used batch 16, length 128 and 6 epochs. The model used was BERT base multilingual cased (12 layers, 12 heads, 110M parameters).

Interesting ideas

• I1. A lexicon of insult words flagged a sentence that explicitly denied the insult, roughly "he was good, not a thief or traitor as they claimed." Surface trigger words break under negation and reported speech.
• I2. Accuracy hid a collapse: the baseline scored around 0.81 accuracy but only about 0.33 macro F1 on imbalanced classes. Reporting both metrics exposed a model that barely detects the minority class.
• I3. Out-of-domain dialect corpora were added one at a time, with a score recorded after each addition. This makes each source's contribution visible instead of just pooling everything.
• I4. Extra corpora were justified partly by their reported inter-annotator agreement. Agreement served as a quality gate for borrowed labels.
• I5. Pretrained BERT embeddings feeding a light BiLSTM head beat plain BERT fine-tuning here. A cheap sequential head on top of rich representations can be a strong combination.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): prag should check whether an evaluative predicate falls under negation or is attributed to another speaker before assigning it to the speaker. Such denial sentences make a ready-made test set.
• I2 (learn): every evaluation should report per-class or macro scores next to overall accuracy. A majority-class predictor should serve as a negative control, so that rare labels or relations cannot be silently ignored.
• I3 (tree): when mixing treebanks or converted UD dialects for gparse or the perceptron parser, add them incrementally and keep an ablation table. That shows which sources actually help.
• I4 (learn): silver data could likewise be admitted or weighted by how many witness parsers agree, with low-agreement items held back.
• I5 (tree): gparse could compare a small recurrent or biaffine head over frozen transformer features against full fine-tuning, to trade cost against accuracy.