📄 AyahVerse at MAHED Shared Task: Fine-Tuning ArabicBERT with Preprocessing for Hope and Hate Detection

Summary

The authors submitted a system to the MAHED Shared Task 2025, competing on its Subtask 1. The task asks systems to label Arabic social media posts about the Middle East conflict as hope, hate, or not_applicable. They used the organisers' dataset of 9843 posts and compared two designs built on ArabicBERT (asafaya/bert-base-arabic):

• a single three-way fine-tuned classifier;
• a stacked system in which two binary ArabicBERT models (hope vs. not_applicable, hate vs. not_applicable) pass their probabilities to a logistic regression meta-classifier.

Both designs were tested with several Arabic-specific preprocessing pipelines and several ways of handling class imbalance. The submitted system was weak and placed last of 25 teams. It used a deeper classification head, minimal preprocessing and few epochs. After the deadline, the authors switched to a plain linear head, Arabic letter normalisation, diacritic and tatweel removal, retained stopwords and weighted loss. With these changes the single classifier reached about 0.60–0.63 macro-F1. One stacked variant with emojis stripped scored 0.91 on test but only 0.59 on validation, so that result needs cautious reading. The paper is mainly useful as a practical record of which preprocessing and imbalance choices helped or hurt.

Key points

• Dataset: MAHED 2025 with 6890 train, 1476 validation and 1477 test posts. The classes are highly imbalanced, and not_applicable dominates.
• The official submission scored macro-F1 0.376 on validation and 0.465 on test. The leaderboard lists it at 0.48 and rank 25/25.
• Imbalance handling for the single classifier (macro-F1, validation/test):
  • no strategy: 0.578/0.608;
  • oversampling: 0.252/0.244;
  • undersampling: 0.236/0.231;
  • class-weighted loss: 0.603/0.635.
  • Training F1 reached 0.98 under both oversampling and weighted loss.
• Training setup: AdamW, CrossEntropyLoss, 8 epochs (up from 3), max length 128, batch size 16, learning rate 2e-5, on Colab T4/L4/A100 GPUs.
• Stacked ensemble results (validation/test):
  • emojis converted to Arabic words: 0.60/0.63;
  • emojis removed: 0.59/0.91;
  • variant with not_applicable split into two halves, one per binary model: 0.58/0.63 with emojis and 0.59/0.65 without.

Interesting ideas

• I1. Turning emojis into dictionary words can flip the signal: a laughing emoji maps to "laughing" (positive), yet it mostly appears in sarcastic hateful or neutral posts. Dropping the emoji worked better than naive translation.
• I2. To stop a dominant "neither" class from swamping two binary experts, the majority class can be split into disjoint halves, one per expert, with a small meta-classifier combining their probabilities.
• I3. Reporting train, validation and test scores side by side for every imbalance strategy made memorisation obvious: oversampling reached 0.98 on train but about 0.25 on held-out data.
• I4. Removing stopwords hurt, because function words carry meaning; restoring them was one source of the gains.
• I5. The 0.59 validation vs. 0.91 test gap from a single preprocessing toggle is a warning that one split can reward a lucky configuration.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): irony detection should treat emoji and similar markers as context-dependent cues whose polarity is resolved against the propositional content, not as fixed lexicon sentiment.
• I2 (learn): when training several narrow detectors on silver data (e.g., per speech act), partition the shared negative pool so each detector sees balanced, non-overlapping negatives.
• I3 (learn): every training-recipe ablation should log train next to held-out scores, so that a "win" coming from duplication or leakage is caught automatically.
• I4 (tokens): never strip function words or particles in preprocessing, since they anchor UD relations such as case, mark and neg, which Mova's no-bag-of-words design depends on.
• I5 (learn): accept a preprocessing change only if it improves several held-out sets and passes negative controls, not one test file.