📄 Evaluating Large Language Models on Sentiment Analysis in Arabic Dialects

Summary

The paper asks how well open large language models classify sentiment in dialectal Arabic, a setting with far fewer resources than Modern Standard Arabic (MSA). To compare varieties fairly, the authors built a small parallel corpus. They cleaned the SemEval-2016 ABSA-Hotels reviews into a balanced set of 538 MSA sentences, machine-translated them into Saudi dialect and Darija with NLLB-200, and had native speakers check the translations. They then fine-tuned Qwen2.5-7B-Instruct, LLaMA-3 and DeepSeek-R1 with 4-bit LoRA and crossed every input variety with prompts written in English, MSA or the input's own dialect.

The best macro F1 was 79%, from Qwen2.5 on Darija with an MSA prompt. DeepSeek did best when the prompt dialect matched the input, reaching 71% on Saudi. LLaMA-3 was the most stable across prompt types. English prompts made Qwen2.5 collapse.

Error analysis shows that neutral and negative sentences are the hardest. Politeness, understatement and sarcasm push predictions toward positive, and dialect-aware prompting reduces this bias. The work matters because it gives a controlled, parallel benchmark for comparing prompt–input alignment across Arabic varieties.

Key points

• Data: ABSA-Hotels (SemEval-2016, reviews from Booking.com and TripAdvisor).
  • Normalised with the ruqiya library and deduplicated.
  • Mixed-polarity items were relabelled, leaving 538 balanced sentences.
  • Each sentence was translated to Saudi and Darija via NLLB-200 and validated by native speakers.
• Training setup:
  • LoRA adapters sit on the query, key, value and output projections, with 4-bit quantization.
  • Training used the Hugging Face Trainer, an 80/20 stratified split, and checkpoint selection by evaluation loss.
  • Prompts had three examples, one per sentiment class.
• DeepSeek-R1 (best with matched prompts):
  • Saudi input with a Saudi prompt: 72% accuracy, 71% macro F1, versus 54% F1 with an MSA prompt.
  • Darija input with a Darija prompt: 64% F1.
  • MSA input: 69% F1 with an MSA prompt and 67% with an English prompt.
• Qwen2.5 (best with MSA prompts):
  • Darija input with an MSA prompt: 81% accuracy, 79% F1.
  • Saudi input with an MSA prompt: 74% F1, versus 64% with a Saudi prompt.
  • English prompts dropped it to roughly 24–28% F1.
• LLaMA-3: reached 75% macro F1 on Darija with an MSA prompt and stayed consistent across prompt variants.

Interesting ideas

• I1. Building one parallel corpus by translating a single labelled source into several varieties and having humans verify it removes data differences from cross-variety comparisons.
• I2. A full grid of instruction language against input variety exposes model-specific sensitivities that a single prompt would hide; here one model gained from matching while another preferred MSA.
• I3. Reporting per-class F1 before and after an intervention, not only an overall score, reveals where a change helps, such as the gains on neutral and negative classes.
• I4. The error analysis traces false positives and false neutrals to polite wording, adequacy statements, understatement and sarcasm, meaning pragmatically encoded negativity rather than lexical cues.
• I5. Prompts contain exactly one exemplar per class, giving a balanced and minimal few-shot setup that avoids biasing the model toward any label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Parallel English variants (dialectal, informal, formal) of a fixed sentence set could show where the perceptron parser and gparse lose LAS purely because of register.
• I2 (learn): When using LLM teacher tiers for silver data, sweep instruction phrasing per text genre and record per-teacher sensitivity before trusting their labels.
• I3 (learn): Evaluations of repair rules or new training rounds should log per-label deltas so that gains on the majority class do not mask regressions on rare labels.
• I4 (frames): The prag crate's implicature and irony components could flag hedged complaints and "it was acceptable" style statements as a test suite, since surface polarity misleads there.
• I5 (answer): Readers asked to judge character attitude in fables could use class-balanced exemplar sets to check that they do not default to the positive reading.