📄 SDNLP at AMIYA 2026: Syrian Arabic Dialect Modeling with LoRA

Summary

Large language models are trained mostly on Modern Standard Arabic (MSA), so when asked for spoken dialect they tend to drift back into the formal register. Built for the AMIYA 2026 shared task's Syrian Arabic track (Open Track), this system approaches the problem as dialect modeling rather than factual QA. It favors natural colloquial output over literal faithfulness, in line with the AL-QASIDA evaluation framework.

The authors fine-tuned the base Meta-Llama-3.1-8B with LoRA adapters on about 30,000 professionally dubbed TV dialogue turns supplied by the company NIS. Each turn is aligned across English, MSA and Syrian Arabic. At inference they added prompt constraints that require exactly one Syrian Arabic sentence with no explanations.

In the official human evaluation, fluency (2.928) was far higher than adequacy (1.124). This reflects the chosen trade-off. Side-by-side outputs show the adapted model giving shorter, more conversational replies than the base model. The work suggests that light adaptation on high-quality, register-aligned data can shift a model toward a spoken variety at low cost.

Key points

• Backbone: Meta-Llama-3.1-8B (base, not instruct), frozen. LoRA settings:
  • adapters on q_proj, k_proj, v_proj and o_proj;
  • rank 16, alpha 32, dropout 0.05;
  • 1.5 epochs with AdamW at lr 2e-4 and effective batch 32 (4 × 8 accumulation);
  • 8-bit loading with FP16 mixed precision.
• Data: about 30k dubbing utterances in a table with the fields character, gender, english, arabic_msa and arabic_syrian. Average lengths are roughly 9.6 words (Syrian), 8.8 (MSA) and 9.9 (English). No official task data or other corpora (e.g., UFAL) were used.
• Human evaluation: Adequacy 1.124, Fluency 2.928.
• Automatic evaluation (chrF/chrF++ per AL-QASIDA):
  • ADI2: 0.280;
  • DA→ENG: 36.211;
  • ENG→DA: 14.813;
  • DA→MSA: 6.743;
  • MSA→DA: 15.609.
• The single-sentence output constraint came from informal preliminary observation, not an ablation. Qualitative comparison rests on four example prompts run against the base model with identical prompts.

Interesting ideas

• I1. Training on three-way aligned text (source language, formal register, colloquial register) lets the model see register contrast directly rather than infer it.
• I2. Keeping one frozen backbone and swapping small per-variety adapters makes dialect extension cheap and modular.
• I3. Hard output-shape constraints in the prompt (one sentence, no meta-commentary) made model outputs shorter and more predictable.
• I4. Scoring fluency and adequacy as separate axes exposes a deliberate trade-off that a single score would hide.
• I5. Running the base and adapted model under identical prompts isolates the effect of adaptation from prompt wording.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): Generation could keep parallel formal/plain/colloquial realizations of the same meaning graph and learn register choice as a controllable switch instead of a side effect.
• I2 (tree): gparse could carry per-genre or per-UD-dialect adapters (e.g., spoken vs. written treebanks) on a shared encoder, retraining only the small part when a new variety appears.
• I3 (learn): LLM teacher tiers could receive strict single-answer formats so silver labels parse reliably and verbose drift is cut before it enters training data.
• I4 (learn): Evaluation of generated or repaired outputs should report form-quality and meaning-preservation separately, so a fluent but wrong output is not counted as progress.
• I5 (learn): Each trained component could be checked against its untrained baseline on the same inputs, serving as a simple negative control for claimed gains.