📄 TranTranUIT at MAHED Shared Task: Multilingual Transformer Ensemble with Advanced Data Augmentation and Optuna-based Hyperparameter Optimization

Summary

The paper addresses Sub-task 1 of the MAHED 2025 Shared Task, which asks systems to label Arabic social media posts as hate, hope, or not_applicable. Three things make this hard: diglossia between MSA and regional dialects (Egyptian, Gulf, Levantine, Maghrebi), noisy informal spelling, and skewed labels, with not_applicable making up over half of the 6,890 training posts. The official metric is therefore macro-F1, and dev/test labels are available only through a blind leaderboard.

The authors fine-tune three encoders with different pre-training backgrounds:

• AraBERTv2, for formal MSA;
• AraBERT-Twitter, for informal and dialectal text;
• XLM-RoBERTa, for multilingual coverage.

Their class probabilities are averaged with equal weights. Training data is expanded by a mix of back-translation (70%) and EDA (30%), and character-level noise is then added. Optuna tunes the main training hyperparameters.

The ensemble reaches 0.65 macro-F1 on the test set, compared with 0.61 for the best single model. The authors attribute most of the gain to better recall on the minority hope class. The result is a cheap, simple recipe for noisy, imbalanced, dialect-rich classification.

Key points

• Training data: 1,301 hate, 1,892 hope and 3,697 not_applicable posts. The dev set (1,476 posts) and test set (1,477 posts) are evaluated only via leaderboard submissions, and all tuning uses the training set alone.
• Test macro-F1: AraBERTv2 0.60, AraBERT-Twitter 0.61, XLM-RoBERTa 0.60, uniform soft-voting ensemble 0.65. The authors say the system beats mBERT and Arabic LLaMA-based models but give no numbers for those baselines.
• Augmentation ablation: no augmentation 0.62, EDA only 0.59, back-translation only 0.60, noise only 0.59, full pipeline 0.65. The prose calls the single methods small gains, but the table shows them below the no-augmentation baseline.
• Pipeline details:
  • back-translation chain Arabic→English→French→Arabic;
  • EDA with synonym replacement, insertion, swap and deletion;
  • normalization: diacritic removal, Alef unification, elongation stripping, removal of non-Arabic symbols and emojis;
  • Optuna over 20 trials (learning rate, batch size, weight decay, dropout) with early stopping;
  • PyTorch 2.2 and HuggingFace Transformers 4.39 on Kaggle P100 GPUs.
• Stated limitations: no significance testing, back-translation depends on third-party APIs, and the combined augmentation costs more compute than any single method.

Interesting ideas

• I1. Choose ensemble members for the language register each one covers (formal, informal, multilingual), not just for architectural variety, so their errors are less correlated
• I2. Averaging probabilities rather than counting votes lets one model's confident prediction for a rare class survive when the other models lean toward the majority class
• I3. Character-level noise alone gave nothing; it helped only when applied on top of meaning-preserving and lexical variation
• I4. The ablation shows individual augmentations can fall below the no-augmentation baseline while their combination wins, so components must be tested both alone and together
• I5. Tuning only on training data and treating the leaderboard as a sealed test keeps reported scores honest under blind conditions

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): pick the silver-data witness parsers so they were trained on different registers (news, web, dialogue), which makes their agreement stronger evidence of a correct tree.
• I2 (tree): combine gparse and perceptron parser scores per arc and label instead of taking majority-vote heads, so rare relations are less likely to be overwritten by frequent ones.
• I3 (tokens): when hardening the tokenizer and tagger against typos, elongations and casing slips, add the noise to already paraphrased sentences rather than to the clean originals only.
• I4 (learn): evaluate each self-induced repair rule or data source both in isolation and in the full mix, always keeping the untouched baseline as a negative control.
• I5 (answer): keep a held-out set of school-course tests and fable QA that is never used for tuning, so reported accuracy reflects real generalization.