📄 AraSentEval 2026: A Shared Task on Sentiment Analysis and Swapping in Arabic

Summary

AraSentEval is a shared task held at OSACT7 (LREC 2026). It targets two weak spots in Arabic sentiment work. The first is that most benchmarks cover MSA or only a few dialects. The second is that controllable sentiment generation is barely studied. The organisers set up two subtasks:

• Subtask 1 is three-way polarity classification over four dialects. It uses MDS-3, a parallel set of hotel-review sentences built from ADOR, translated and labelled by native speakers.
• Subtask 2 asks systems to rewrite a sentence so its polarity flips while its content stays the same. Its data combine the hand-built MA'AKS pairs with LLM-assisted dialect pairs. The test portion contains only dialect pairs.

Scoring ran on CodaBench. Subtask 1 used Macro F1. Subtask 2 used a classifier-based style accuracy together with BLEU and chrF against references. Fifteen teams took part. Classification turned out close to solved, with a top Macro F1 of 0.9423. Sentiment swapping proved much harder, with best style accuracy around 0.76. The swap results show a real tension between changing the polarity and keeping the original content. This makes the benchmark useful for testing semantics-aware generation in dialects.

Key points

• MDS-3: 3,000 sentences labelled positive, negative or neutral.
  • It is balanced across Moroccan, Egyptian, Jordanian and Saudi Arabic, and the sentences are parallel across dialects.
  • Train and test are split 80/20 by sentence ID, so that translations of one sentence never land on both sides.
• MA'AKS: 5,000 manually curated MSA positive/negative pairs, divided 4,000/500/500.
  • It was previously evaluated with AceGPT, JAIS and Llama-3 under zero-shot, few-shot and fine-tuned setups.
• Final Subtask 2 data:
  • 6,263 training pairs and 1,315 development pairs, mixing MSA with Egyptian, Gulf, Levantine and Moroccan.
  • 646 test pairs, all dialectal.
• Participation: 15 teams took part and 9 wrote system papers.
• Subtask 1 winner: musj1984 reached Macro F1 0.9423. It used a Mixture of Experts that gates two QLoRA-tuned encoders (AraBERT v2 and BERT-Large Arabic), trained on LLM-augmented data.
• Subtask 2 winner: Yumnahamdy, with an AraBART + AraT5v2 ensemble and dialect-aware augmentation. It scored style accuracy 0.7554, BLEU 43.00 and chrF 65.36.
• Subtask 2 failure case: VGU-M.Tech-AI reached only 0.2368 accuracy, yet its BLEU of 30.79 was higher than some stronger systems.

Interesting ideas

• I1. Splitting data by the ID of the source sentence keeps all parallel variants in one fold, which blocks hidden leakage whenever data contain translations or paraphrases of each other.
• I2. The test set was deliberately limited to the harder varieties (dialects), while the easier MSA appeared only in training, so the score measures transfer rather than familiarity.
• I3. Swap outputs were judged on two separate axes, whether the target attribute was reached and how much of the source survived. A system that barely edits its input can therefore score well on overlap yet fail on attribute change.
• I4. The organisers paired a recognition task with an inversion task over the same phenomenon, polarity. Being able to flip a property shows whether a system knows where in the sentence that property lives.
• I5. The winning classifier used a learned gate that weights two fine-tuned models per input, instead of a fixed vote.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when silver data include several renderings of one source sentence, assign folds by source ID so that evaluation scores are not inflated.
• I2 (tree): hold out whole genres or UD dialects (e.g., web text, transcribed speech) as test-only, to see whether gparse and the perceptron parser generalise beyond their home treebank.
• I3 (back): score text → graph → text output on both "intended change realised" and "content kept", and add a copy-the-input baseline as a negative control.
• I4 (frames): build minimal pairs that negate or invert evaluative predicates, and check that prag's sentiment and irony analysis changes only in the frame slots that were actually edited.
• I5 (learn): a small gate over the witness parsers could learn which witness to trust for each construction, giving better-calibrated silver agreement than plain majority.