📄 BDSI at AraSentEval Shared Task : A Multi-Transformer Contrastive Learning for Arabic Dialect Sentiment Analysis

Summary

The paper describes a system for Subtask 1 of the AraSentEval 2026 shared task: three-way sentiment classification (positive, negative, neutral) of hotel reviews written in several Arabic dialects. Dialectal Arabic has no standard spelling, mixes with MSA, and has few labelled resources. The authors fine-tune two pretrained Arabic encoders, AraBERTv2 and CAMeLBERT-Mix, each with its own preprocessing. Training combines class-weighted, label-smoothed cross-entropy with a supervised contrastive term computed on a projected [CLS] embedding. At inference, the two models' softmax outputs are averaged. A small set of regex rules then corrects dialect-specific cases the models tend to miss. On a stratified validation split, each model scores about 0.93 macro F1. The ensemble reaches 0.83 macro F1 on the official test set. The authors attribute the gap to dialect variety in the unseen data. The work shows a compact, reproducible recipe for low-resource dialect classification, trained only on the provided data and run on Google Colab.

Key points

• Data: 1,731 hotel reviews in Darija, Saudi, Jordanian and Egyptian dialects. Labels are positive 609, negative 657 and neutral 465. The data was split 85/15 with stratification, with no external data or augmentation.
• Models: aubmindlab/bert-base-arabertv2 and CAMeL-Lab/bert-base-arabic-camelbert-mix. Each has a two-layer GELU MLP classifier (dropout 0.3) and a 128-dimensional projection head for the contrastive loss.
• Loss: cross-entropy (label smoothing 0.1, class weights) plus 0.2 × supervised contrastive loss with temperature 0.07.
• Training: AdamW with learning rate 2e-5 for the backbone and 1e-5 for the heads, batch size 16, maximum length 128, and early stopping on validation macro F1 (patience 5).
• Results: validation macro F1 was 0.9355 for AraBERTv2 and 0.9319 for CAMeLBERT. The ensemble scored 0.8301 macro F1 and 0.8301 accuracy on the test set.

Interesting ideas

• I1. Each encoder gets the preprocessing that fits its pretraining: the official ArabertPreprocessor for AraBERTv2, and nearly raw text for CAMeLBERT so its dialect cues survive. This avoids one shared normalization that suits neither model.
• I2. An auxiliary supervised contrastive loss, applied through a separate projection head, pulls same-label examples together in embedding space without changing the main classifier.
• I3. The ensemble averages full probability distributions from two complementary models instead of taking hard votes, so it keeps their uncertainty.
• I4. Post-hoc rules fire only on specific discourse patterns and only when the model's initial prediction matches the rule's condition. The patterns include a decline over time ("was excellent… became acceptable"), stated intent to return, and doubts about a review's authenticity.
• I5. The roughly 0.10 drop from a random in-distribution validation split to the official test set warns that such splits overstate robustness on varied input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Keep raw and normalized forms side by side, and give each downstream component (tagger, lemmatizer, gparse) the variant it was trained on, rather than forcing one normalization on all of them.
• I2 (learn): When training gparse, a contrastive term over arc or relation-label embeddings could sharpen distinctions between often-confused UD relations without altering the scoring head.
• I3 (learn): Silver-data agreement among witness parsers could average per-arc probability distributions, flagging low-margin arcs for the LLM teacher instead of discarding them.
• I4 (frames): These patterns map naturally to explainable pragmatic cues: a temporal-contrast frame marking a state change, a commitment speech act, and a meta-comment that undercuts the text's sincerity.
• I5 (learn): Evaluate on held-out genres or sources as a standard control, so scores from same-distribution dev sets are not mistaken for generalization.