Elamur Research⌕
☰
home / papers / 2026 / NUS-IDS at AMIYA/VarDial 2026: Improving Arabic Dialectness in LLMs with Reinforcement Learning

NUS-IDS at AMIYA/VarDial 2026: Improving Arabic Dialectness in LLMs with Reinforcement Learning

Summary

The paper addresses a known weakness of current LLMs: when asked for Dialectal Arabic (DA), they tend to answer in Modern Standard Arabic (MSA). The NUS-IDS team entered the Closed Data track of the AMIYA shared task at VarDial 2026, which covers Egyptian, Moroccan, Palestinian, Saudi and Syrian Arabic. Using only the organisers' training data, they instruction-tuned Llama-3.1-8B-Instruct into a dialect classifier and also trained a dialect translator. They then combined AL-QASIDA task prompts with sampled AMIYA sentences and collected answers from several open and proprietary LLMs. The classifier labelled each answer. Mismatched answers were pushed through the translator and checked again, which produced preferred and dispreferred responses for DPO training. On FLORES+ and on the official test, the RL-tuned models became far more dialectal. Their translation quality (ChrF++, SpBLEU), however, fell clearly below the base models. The work matters as a cheap way to build alignment data from a classifier. It also shows plainly that rewarding a single property can harm overall output quality.

Key points

  • Dialect classifier (FT-Llama): trained on about 300K AMIYA sentences, about 50K per class, with MSA included as a class.
    • It reaches F1 from 0.83 (Saudi) to 0.94 (Palestinian).
    • The NADI baseline (MARBERT, 18 dialects, no MSA) scores only 0.59–0.83 on the same held-out set.
  • Training data:
    • The translator used about 697K sentences; NLLB-200 filled in missing MSA/English sides of the parallel data.
    • The preference set has about 9.6K pairs. It was built from about 24 AL-QASIDA prompts per dialect and about 1000 sentences per dialect with a dialectness score above 0.36, plus MSA→DA translation prompts.
  • Training setup: LoRA via LlamaFactory, 3 epochs, default hyperparameters, DPO for the RL step, on two 48GB A6000 GPUs.
  • Validation trade-off: llama+RL raised AvgALDi from 0.1659 to 0.6949, while chrF++ dropped from 32.09 to 23.03.
    • Translation fine-tuning helped qwen-T but not llama-T.
    • Saudi scores look anomalous because the Saudi part of FLORES is close to MSA.
  • Official results:
    • The primary run (llama+RL) got ADI2 0.629, above the 0.452 listed for MBZUAI.
    • Its best ChrF++ scores (for example 21.3 DA-ENG) trail the top systems by a wide margin (for example 53.4).

Interesting ideas

  • I1. A trained classifier can act as the judge that sorts many generators' outputs into preferred and rejected answers, and a repair model (here a translator) can turn failures into new positives after a second check, giving preference data without human labelling.
  • I2. Rewarding only one target property (dialect) let the models drift: they produced repeated characters, mangled numbers and names, and Latin-script dialect where English was requested, so a preference signal needs a quality term alongside the target trait.
  • I3. Including the "standard" variety (MSA) as its own class in a multiple-choice style classifier made it much stronger than a broad off-the-shelf dialect model that lacked that class.
  • I4. An ablation comparing clean parallel data with machine-augmented data (NLLB) separates model effects from noise in synthetic training data.
  • I5. Odd per-subgroup scores (Saudi) were traced to the benchmark's own composition rather than to model skill, a reminder to inspect test sets per slice.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness-parser agreement could choose the winning parse, while a repair rule fixes losing parses and re-validates them, yielding preference pairs for training the parsers.
  • I2 (learn): Self-play and repair rules should score candidates on several criteria at once, and negative controls should check that improving one metric does not quietly break another.
  • I3 (tree): When converting between UD dialects or treebank conventions, a classifier that explicitly includes the default convention could flag which scheme an input sentence follows before conversion.
  • I4 (learn): Train each LLM-teacher tier with and without its silver portion to measure whether the teacher data helps or hurts specific parser components.
  • I5 (answer): Break school-test and fable-QA scores down by question type and source, and check whether a suspiciously easy slice is simply closer to the training distribution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining Semitic languages

 Institutions

🎓National University of SingaporeSingapore · Singapore61 works · 28,316 citations🎓Nanyang Technological UniversitySingapore · Singapore56 works · 22,783 citations

Also on the byline: Institute of Data Science.

Cite

Cite as Sujatha Das Gollapalli, Mouad Hakam, Mingzhe Du and See-Kiong Ng. 2026. NUS-IDS at AMIYA/VarDial 2026: Improving Arabic Dialectness in LLMs with Reinforcement Learning. Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.30

@inproceedings{gollapalli2026nus,
  title = {{NUS-IDS at AMIYA/VarDial 2026: Improving Arabic Dialectness in LLMs with Reinforcement Learning}},
  author = {Sujatha {Das Gollapalli}, Mouad Hakam, Mingzhe Du and See-Kiong Ng},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.30},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.30}
}

References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  2. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. NADI 2024: The fifth nuanced Arabic dialect identification shared task. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 709–728, Bangkok, Thailand. Association for Computational Linguistics. find
  3. Abdulkareem Alsudais, Wafa Alotaibi, and Faye Alo-mary. 2022. Similarities between arabic dialects: Investigating geographical proximity. Information Processing Management, 59:102770. our card
  4. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  5. Marta Costa-jussa, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Gonzalez, Prangthip Hansanti, John Hoffman, and Jeff Wang. 2024. Scaling neural machine translation to 200 languages. Nature, 630. Md. our card
  6. Arid Hasan, Prerona Tarannum, Krishno Dey, Imran Razzak, and Usman Naseem. 2024. Do large language models speak all languages equally? a comparative study in low-resource settings. Preprint, arXiv:2408.02237. ↗
  7. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. Preprint, arXiv:2410.15553. ↗
  8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR 2022. our card
  9. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. our card
  10. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Comput. Surv., 57(6). find
  11. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics. find
  12. Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, and Jungwook Choi. 2024. Improving conversational abilities of quantized large language models via direct preference alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11346–11364, Bangkok, Thailand. Association for Computational Linguistics. find
  13. Zhiheng Yao, Changsong Yu, and 1 others. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 658–671. our card
  14. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155. ↗ our card
  15. Juan Antonio Perez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena, Miquel Esplà-Gomis, Aaron Galiano Jimenez, Antoni Oliver, Claudi Aventín-Boya, Alejandro Pardos, Cristina Valdés, Jusèp Loís Sans Socasau, and Juan Pablo Martínez. 2024. Expanding the FLORES+ multilingual benchmark with translations for Aragonese, aranese, Asturian, and Valencian. In Proceedings of the Ninth Conference on Machine Translation, pages 547–555, Miami, Florida, USA. Association for Computational Linguistics. find
  16. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. our card
  18. Nathaniel Robinson, Shahd Abdelmoneim, Anjali Kantharuban, Otba Alsboul, Salima Lamsiyah, Kelly Marchisio, and Kenton Murray. 2026. AMIYA shared task: Arabic Modeling In Your Accent at Var-Dial 2026. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, Rabat, Morocco. Association for Computational Linguistics. find
  19. Nathaniel Romney Robinson, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. 2025. AL-QASIDA: Analyzing LLM quality and accuracy systematically in dialectal Arabic. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22048–22065, Vienna, Austria. Association for Computational Linguistics. find
  20. Shubham Vatsal and Harsh Dubey. 2024. A survey of prompt engineering methods in large language models for different nlp tasks. Preprint, arXiv:2407.12994. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gollapalli-2026-nus-ids-amiya-vardial-2026/