📄 iWAN-NLP at AHaSIS 2025: A Stacked Ensemble of Arabic Transformers for Sentiment Analysis on Arabic Dialects in the Hospitality Domain

Summary

The paper addresses three-way sentiment classification (positive, negative, neutral) of Arabic hotel reviews that mix Modern Standard Arabic with regional dialects, the problem posed by the AHaSIS 2025 shared task. The author used only the organizers' training split, with no external data and almost no text cleaning. Three Hugging Face encoders with different regional strengths were fine-tuned: MARBERTv2 (broad social-media Arabic), SaudiBERT (Saudi dialect) and DarijaBERT (Moroccan Darija). Each was trained under 5-fold stratified cross-validation with label smoothing, early stopping on validation F1, learning-rate warmup and FP16. At test time, the logits of the five fold models of each architecture are averaged, the three per-architecture averages are averaged again, and argmax gives the label. The title says "stacked", but the combination is plain soft averaging, not a learned meta-classifier. The system reached macro F1 0.8055 on the official test set, which ranked first, and 0.8513 in cross-validation. The result suggests that pooling dialect-specialized encoders is a cheap and strong recipe for dialect-heavy classification.

Key points

• Data: AHaSIS 2025 hotel reviews with three labels and official train/test splits. No augmentation or outside corpora were used. Preprocessing was limited to mapping labels to IDs and running each model's own tokenizer.
• Base models: UBC-NLP/MARBERTv2, faisalq/SaudiBERT and SI2M-Lab/DarijaBERT, each with an AutoModelForSequenceClassification head and a custom Hugging Face Trainer.
• Training setup:
  • AdamW, lr 2e-5 with warmup, batch size 8;
  • up to 6 epochs per fold, early stopping on F1;
  • nn.CrossEntropyLoss with label smoothing 0.1, FP16;
  • a single Google Colab GPU.
• Scores: test macro F1 0.8055 (first place). Cross-validated macro F1 was 0.8513 with accuracy 0.8628, so the test score was roughly 4.6 points lower.
• Error analysis: the main confusion was between neutral and negative. Likely causes listed are sarcasm, implicit sentiment, code-switching, annotation artifacts, and texts tagged as Darija that are actually mixed with MSA or other dialects.

Interesting ideas

• I1. Two-level score averaging reuses all cross-validation checkpoints at inference: fold models are averaged first, then architectures, which smooths out run-to-run variance without retraining on the full data.
• I2. The ensemble's diversity comes from encoders pretrained on different language varieties, each fed through its own tokenizer, rather than from random seeds of one model.
• I3. Out-of-fold predictions were used to measure the ensemble during development, giving an estimate on data no model in that fold had seen.
• I4. The error analysis notes that a plain factual report of a service failure carries negative sentiment without any negative words. Such cases are where neutral and negative get confused.
• I5. The analysis also flags that metadata labels can be wrong: some texts tagged as Darija were really mixed dialect or MSA, which hurt the Darija specialist model on those items.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could average arc and label score matrices from fold-trained checkpoints before decoding, which keeps the output a valid tree, unlike voting on finished trees.
• I2 (learn): witness parsers trained on different genre treebanks (web, spoken, academic) would make agreement-based silver data more trustworthy than witnesses that share one training domain.
• I3 (learn): self-induced repair rules could be induced on some folds and scored only on the held-out fold, which works as a built-in negative control against overfitting.
• I4 (frames): an implicature rule could turn a reported failed expected state ("the AC did not work") into an explicit, traceable negative evaluation of the related entity.
• I5 (learn): before routing text to genre- or dialect-specific components, the declared metadata should be checked against the text itself, and mismatches logged as a separate error class.