📄 Lab17 @ Ahasis Shared Task 2025: Fine-Tuning and Prompting techniques for Sentiment Analysis of Saudi and Darija Dialects

Summary

The Lab17 team at Sultan Qaboos University built a baseline entry for the Ahasis 2025 shared task. The task is three-way sentiment classification (positive, negative, neutral) of hospitality-domain texts written in Saudi and Moroccan Darija Arabic, two dialects with few annotated resources. The authors did not propose a new algorithm. Instead, they compared three standard strategies under scarce labeled data:

• Few-shot prompting: GPT-4o with a hand-built prompt containing a few examples.
• Cross-dialect transfer: SODA-BERT, a MARBERT model previously tuned on Omani sentiment, applied without any further adaptation.
• Direct fine-tuning: MARBERT trained on the official Ahasis training split, with a dialect marker token prepended to each input.

Direct fine-tuning won with a macro F1 of 0.75, ahead of SODA-BERT (0.71) and GPT-4o (0.69). The system placed 8th, while the top team reached 0.81. The study matters as a reference point for low-resource dialect work. It shows that a modest encoder fine-tuned on in-task data still beats a large prompted LLM. It also shows that sentiment knowledge transfers reasonably well between related Gulf and Maghrebi varieties.

Key points

• Data: The training set has 430 items per dialect (154 positive, 168 negative, 108 neutral), giving 860 in total. The test set has 216 items, split evenly between Darija and Saudi.
• GPT-4o prompt: It included two example texts per sentiment class per dialect and scored F1 0.69 on the official test.
• SODA-BERT (huggingface.co/mktr/SODA-BERT) was tuned only on Omani data and reached F1 0.71 on Saudi/Darija with no additional training.
• Fine-tuned MARBERT: F1, accuracy, precision and recall all came out at 0.75, with balanced accuracy at 0.746. This compares with an organizer-side baseline of 0.56 (Alharbi et al., 2025).
• Training recipe: AdamW (β₂=0.98), cosine schedule with warmup, label smoothing 0.05, max length 128, fp16, early stopping on macro F1. The paper's hyperparameter reports are internally inconsistent: one passage gives 5e-5 and 8 epochs, while another says validation favored 2e-5 and 5 epochs.

Interesting ideas

• I1. Prepending a dialect identifier token is a near-free way to let one shared model condition on the input's variety, without separate heads or separate models.
• I2. Running an already-adapted neighbouring model off the shelf, before any fine-tuning, gives a cheap measure of how much task knowledge transfers across closely related domains.
• I3. A small fine-tuned encoder beat a strong few-shot LLM on in-task data, which suggests LLMs are better used to supply labels than to serve as the final classifier.
• I4. Reporting balanced accuracy next to macro F1 exposes whether a classifier is quietly favouring the majority classes when the class distribution is uneven.
• I5. Mild label smoothing was used to curb overconfidence when training on only a few hundred examples per variety.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could take a treebank/genre prefix token (EWT vs GUM vs web text) when trained on mixed UD English data, so it can absorb annotation-convention differences instead of averaging them.
• I2 (learn): before retraining on a new text genre, scoring the existing parser and readers on it unchanged would give a transfer baseline, showing how much the new training actually adds.
• I3 (learn): this supports keeping LLM teacher tiers as producers of silver labels that train small explainable components, rather than routing runtime decisions through the LLM.
• I4 (frames): speech-act and irony detectors in prag, where some classes are rare, should report both metrics so skew toward frequent labels gets caught.
• I5 (learn): when training on silver data that several witness parsers only partly agree on, smoothing targets could soften the damage from residual label noise.