📄 AAA at MAHED Text-based Hate and Hope Speech Classification: A Systematic Encoder Evaluation for Arabic Hope and Hate Speech Classification

Summary

The paper describes a system for the MAHED shared task at ArabicNLP 2025, where Arabic social media posts are sorted into three classes: hope, hate, or not_applicable. Arabic is hard for this task because of its rich root-and-pattern morphology, diglossia between MSA and regional dialects, and the frequent use of sarcasm and veiled hostility online. The authors did not build an ensemble. They fine-tuned six transformer encoders one at a time, each with a linear classification head and class-weighted cross-entropy, so they could find the strongest single model for resource-constrained deployment. Three of the encoders are Arabic-specific (MARBERT, AraBERT-Twitter, ALCLAM) and three are multilingual (XLM-RoBERTa, LaBSE, BGE-m3). MARBERT, which was pretrained on dialect-rich social media text, came out on top with a test macro-F1 of 0.707 and similar accuracy, precision and recall. An error analysis showed that most remaining mistakes involve implicit hate, dialectal vulgarity and statements that need outside context. The study is a useful baseline for choosing an encoder for Arabic affective classification.

Key points

• The data consists of Arabic social media posts from the MAHED organisers, split into 6890 training, 1476 validation and 1477 test examples. The labels are hope, hate and not_applicable, and neutral posts make up most of the set.
• Test macro-F1 for each model:
  • MARBERT: 0.707 (accuracy 0.712, precision 0.705, recall 0.710)
  • XLM-RoBERTa: 0.645
  • AraBERT-Twitter: 0.630
  • LaBSE: 0.627
  • BGE-m3: 0.588
  • ALCLAM Base v2: 0.404
• ALCLAM's weak score surprised the authors. One passage of the paper still groups it with the best models, which conflicts with the results table.
• MARBERT was trained with a learning rate of 1×10⁻⁶, batch size 64, 3 epochs with early stopping, warmup ratio 0.1, weight decay 0.01 and a maximum of 170 tokens. Class weighting in the loss was used to handle the imbalance.
• The authors group the main errors into three types:
  • implicit accusation read as hate, for example a post suggesting Messi uses stimulants;
  • dialectal swearing in posts that are otherwise neutral;
  • political remarks that need context to interpret.

Interesting ideas

• I1. Encoders pretrained on the same register as the target data (dialectal social media text) beat larger, broadly multilingual models, so a domain match can matter more than model scale or language coverage.
• I2. Running every candidate as a single model under one identical protocol, with no ensembling, makes the comparison clean and directly tied to what will be deployed.
• I3. The final model is chosen because its precision, recall and accuracy are close to one another, not because it peaks on a single number.
• I4. The error analysis sorts failures into named categories (implicit meaning, dialect or vulgarity, missing context), which tells you where to work next.
• I5. Class weighting in the loss addresses a dataset dominated by one class without changing the data itself.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): When choosing a backbone for gparse, compare candidates on the actual genre Mova reads, such as fables, school texts and word problems, instead of assuming the largest general model wins.
• I2 (learn): Evaluate each witness parser or teacher tier separately under a fixed setup before combining them into silver data, so the contribution of each one is known.
• I3 (learn): Report balanced per-class metrics, for example per dependency relation or per frame role, and prefer components with consistent behaviour, since a skewed one can quietly corrupt later stages.
• I4 (frames): Tag implicature, irony and speech-act errors in prag with similar labels so that self-induced repair rules can be aimed at specific failure types.
• I5 (tokens): Weight rare tags or morphological features during tagger training so that the frequent classes do not crowd them out.