📄 LoveHeaven at MAHED 2025: Text-based Hate and Hope Speech Classification Using AraBERT-Twitter Ensemble

Summary

The paper describes a text-only system for Sub-task 1 of the MAHED 2025 shared task at ArabicNLP 2025. The task is to label short Arabic social-media posts, in MSA or dialect, as hate, hope, or not_applicable, and systems are scored by macro-F1. The authors fine-tune aubmindlab/bert-base-arabertv02-twitter, an AraBERT variant further pretrained on tweets. Before training, they clean the data: they normalize text with ArabertPreprocessor, strip URLs, mentions and non-Arabic characters, and resolve duplicated training items. They then tune hyperparameters with Optuna and train one model per fold of a 4-fold stratified split. At inference, the four models' logits are averaged. The Twitter-adapted model edges out standard AraBERT on validation and test, and the ensemble beats a single model trained on all the data. The system placed 4th on the leaderboard, roughly 0.02 macro-F1 behind the winner. It ran entirely on free Kaggle hardware. The work shows that careful data hygiene, a domain-matched encoder and modest ensembling get close to the top of a noisy, imbalanced, dialect-heavy classification task.

Key points

• Data: 6,890 labeled training samples and 1,476 validation samples whose labels were hidden, so scores came only from the leaderboard; about 320 duplicates appeared in the training set.
• Duplicate policy: identical texts with conflicting labels were all dropped, and identical texts with the same label were reduced to a single copy.
• Tuning: Optuna ran 18 trials at max_length 256 and 30 trials at 128, searching learning rate in [1e-6, 1e-5], 2–5 epochs and batch sizes {8, 16, 32}. The chosen setup was max_length 256, learning rate ≈ 9.74e-6, batch 8 and 4 epochs, trained with AdamW, cross-entropy and linear warmup-decay.
• Results: on validation, AraBERT-Twitter scored macro-F1 0.6563 against 0.6403 for AraBERT; on test, it scored 0.7030 against 0.7017. The final ranking was 4th overall, with accuracy and precision in the top 3.
• Compute: a single Tesla P100 16GB on the Kaggle free tier, which limited how widely hyperparameters could be explored.

Interesting ideas

• I1. Duplicates that disagree on their label are treated as unreliable and discarded entirely, rather than resolved by majority vote, which removes contradictory supervision cheaply
• I2. Fold models are combined by averaging their raw logits rather than voting on final labels, which keeps each model's confidence and smooths out per-fold variance
• I3. The input length limit is chosen from the actual distribution of text lengths and then confirmed experimentally, rather than set to a default
• I4. The encoder is picked to match the register of the data (tweets for informal dialect text), and this alone gave a measurable gain
• I5. Macro-F1 is used as the main metric so that minority classes count as much as the dominant one

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when witness parsers produce identical sentences with conflicting silver trees, drop the item instead of picking a winner, so that only agreed data enters training.
• I2 (tree): gparse could average arc and label scores from several fold-trained checkpoints before decoding, which should give steadier trees on rare constructions.
• I3 (tree): set the gparse context window from measured sentence lengths per treebank, so long sentences are not truncated and compute is not wasted on padding.
• I4 (tokens): pick or adapt the tagger and morphology models to the genre of the input, such as fables or school tests, rather than relying on one generic model.
• I5 (frames): report macro-F1 over speech-act and irony labels, so that rare acts are not hidden behind frequent assertions.