📄 A Gemini-Based Model for Arabic Sentiment Analysis of Multi-Dialect Hotel Reviews: Ahasis Shared Task Submission

Summary

The paper describes a submission to the Ahasis Shared Task, which asks systems to classify Arabic hotel reviews as positive, neutral or negative. The reviews are written in Saudi dialect and Moroccan Darija, two varieties far from MSA, so models trained mainly on MSA handle them poorly. The author does not fine-tune any model. Instead, Gemini Pro 1.5 is called through its API with a few-shot JSON prompt. Before inference, a preprocessing step cleans the text, normalizes elongated spellings, adds a dialect tag to the prompt and marks possible sarcasm.

Gemini was chosen over GPT-4o, LLaMA-3, Claude 3.5, CAMeL, AraBERT and others by testing them on a separate in-house benchmark of YouTube comments. On the official test the system reached F1 0.7361 and ranked 10th. The main errors were neutral factual statements labeled as positive and missed sarcasm. The work matters because it shows that prompt design alone, with no retraining, gives competitive results on low-resource dialectal data.

Key points

• The Ahasis training set has 860 reviews: 336 negative, 216 neutral, 308 positive. The competition evaluation covered 3,000 samples.
• The internal benchmark started as 577 hand-labeled YouTube comments from AJ360 shows. Manual paraphrasing of minority classes grew it to 700. On this benchmark Gemini Pro 1.5 scored 81.46% accuracy and 0.801 Macro-F1, ahead of the other models tested.
• Inference setup:
  • a prompt with 20 examples balanced across dialects;
  • temperature 0 and topP 0.95;
  • all safety filters set to BLOCK_NONE;
  • batch calls run from a spreadsheet Apps Script, with the code released on GitHub.
• Sarcasm flagging combines the ArSarcasm dataset with hand-written rules that look for contradictions.
• Official result: F1 0.7361, balanced accuracy 0.7229, 10th place. The results come from a single run, and no significance test against other teams was possible.

Interesting ideas

• I1. When the model makes errors, add new few-shot examples that target those errors, for example plain factual statements labeled neutral. This fixes the problem without any retraining.
• I2. Detect the language variety first, then pass it as an explicit tag to the downstream model, so later steps know which conventions apply.
• I3. Use contradiction patterns as a sarcasm cue: strong praise followed by a bad consequence (like a room so wonderful the guest could not sleep).
• I4. The main error source was confusing an objective description (good location, cheap price) with the speaker's emotional stance.
• I5. Choose the model by testing it on a different, noisier domain than the target. This checks robustness before committing to a final system.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): LLM teacher prompts could be repaired the same way, by collecting disagreements with the witness parsers and adding contrastive examples of exactly those cases.
• I2 (tree): tag the variety or register of input text (e.g. informal web English vs. edited prose) before parsing, so gparse or the UD dialect-conversion rules can choose the matching settings.
• I3 (frames): the irony detector in prag could check whether an evaluative frame conflicts with an outcome frame that describes harm to the same experiencer.
• I4 (world): store attribute facts about entities separately from the author's attitude, so answers about "what the hotel is like" and "how the writer feels" draw on different stores.
• I5 (learn): keep a domain-shifted held-out set (e.g. comments or forum posts) next to treebank tests, as a check that a gain is not just overfitting to one genre.