📄 A Closed-Track System for Palestinian Arabic in the AMIYA Shared Task

Summary

The authors describe their entry to the closed track of the AMIYA shared task at VarDial 2026. The goal was to teach a small instruction-tuned LLM to generate Palestinian Arabic and to translate between it, MSA and English. They merged Palestinian-labelled material from the Maknuune lexicon, the Shami Twitter corpus and Casablanca speech transcripts. Jordanian and Syrian data (JODA, MASC, other Shami portions) were set aside for diagnostics only. Everything was cast into instruction–response records.

To choose a base model, several candidates were screened on a 500-example dialect-classification probe. Qwen2-1.5B-Instruct was picked and adapted with LoRA, which trained about 0.28% of its weights.

Qualitatively, generation of greetings, discourse particles and idioms often sounded natural. Translation from MSA was weak: the model frequently slid back into MSA or simply echoed its input. A 200-sample LLM-judged audit sorted the failures into five recurring types. The work suggests that cheap adaptation is workable for a low-resource dialect, but also shows where it breaks without more task-aligned data.

Key points

• The Combined Dialect Dataset has 155,299 entries; 48,278 are Palestinian. Training drew solely on the Palestinian portions of Maknuune, Shami and Casablanca, with an 80/10/10 split and seed 42.
• Translation pairs came only from Maknuune and cover four directions: MSA→DA, DA→MSA, EN→DA and DA→EN.
• Model screening on 500 dev samples:
  • TinyLlama-1.1B and Qwen2-1.5B-Instruct both reached 44% accuracy.
  • Qwen scored higher on macro F1 (24.76% vs 20.37%).
  • Qwen was about ten times faster (10.25 vs 1.09 samples/s).
• LoRA settings: r=16, alpha=32, about 4.2M trainable parameters. Training used AdamW with lr 2e-4, batch 8, gradient accumulation 4, 3 epochs, and was capped at 15,000 examples.
• Error distribution in the 200-sample audit:
  • drift to MSA: 31%
  • instruction violation: 24%
  • input copying: 18%
  • semantic shift: 15%
  • hallucination: 12%

Interesting ideas

• I1. Candidate models are ranked on one small fixed probe by macro F1, with throughput as the tie-breaker; this is a cheap way to prune many options before spending compute on training.
• I2. Neighbouring-variety corpora are kept out of training but retained as diagnostic material, so their influence can be studied without contaminating the target system.
• I3. "Input copying" is tracked as its own error class, which separates a model that echoes its source from one that genuinely mistranslates.
• I4. An LLM judge assigns one of three labels (correct, partially correct, incorrect) plus a short justification, and the error taxonomy is then built inductively from the non-correct cases.
• I5. Memorization is checked by looking for idiom variants that never appeared verbatim in training, as evidence that productive patterns were learned.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Witness or student parser variants could be screened on a fixed small dev slice using LAS first and speed second, before full training runs.
• I2 (tree): During UD dialect conversion, treebanks of other English varieties or genres could serve as diagnostic-only held-out sets that reveal drift in gparse.
• I3 (back): The text → graph → text compressor and the translation path could flag outputs nearly identical to their input as a negative control, rather than counting them as successes.
• I4 (learn): LLM teacher tiers could return a graded verdict with a reason, and clusters of those reasons could seed self-induced repair rules.
• I5 (answer): Readers could be tested on fables or word problems rephrased into unseen surface forms, to confirm they rely on structure and not on memorized strings.