Elamur Research⌕
☰
home / papers / 2026 / AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026

AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026

Summary

Large language models handle Modern Standard Arabic (MSA) reasonably well, but they tend to default to it even when users write in or ask for a dialect. This disadvantages the many Arabic speakers whose everyday language is dialectal. AMIYA, held at VarDial 2026, is the first shared task devoted to dialectal Arabic language modeling. It covers the varieties of Morocco, Egypt, Palestine, Syria and Saudi Arabia. The organizers offered three tracks (closed data, closed model, open) and a training pool assembled from 17 corpora. Systems were scored with the AL-QASIDA methodology: ADI2 for dialectal fidelity, chrF++ for translation in four directions, and human ratings of adherence and fluency. Six teams submitted 45 systems. Fine-tuning on translation and reinforcement learning for dialectness both helped. The strongest approach combined continual pre-training and supervised fine-tuning across 12 candidate LLMs, then kept the best performers. Human judges found that high dialect scores can hide incoherent output built from dialectal vocabulary. Lexical knowledge therefore does not equal conversational competence. The task provides a benchmark and practical lessons for future dialect-aware LLMs.

Key points

  • Tracks:
    • The closed data track allowed any open-source LLM but only organizer-provided data. This data came from 17 sources, including MADAR 26, Atlaset (155M Moroccan tokens), DODa, the Shami Corpus, Maknuune, SauDial, JODA and UFAL.
    • The closed model track required training from scratch.
    • The open track allowed any open model and any non-test data.
  • Evaluation data: Prompts were drawn from Palm, FLORES-200, Arena-Hard and MADAR-26.
    • Palm sentences were kept only if their ALDi score exceeded 0.5. This yielded 649 Moroccan, 283 Egyptian, 419 Palestinian, 181 Syrian and 141 Saudi items.
    • 220 Arena-Hard English prompts were rewritten to request dialectal answers. Half were rewritten by hand and half with Gemini, followed by manual checking.
  • Human evaluation: Native speakers rated 100 outputs per team's primary system. Maastricht University's (UM) primary submission received the top human scores. Even the best models were judged understandable but likely non-native in fluency.
  • Limitations noted by the organizers:
    • AL-QASIDA was hard to set up and required GPUs, so they suggest a "lite" version.
    • Some FLORES-200 dialect subsets, notably Saudi, are too close to MSA, which makes results based on them questionable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.1

@inproceedings{robinson2026amiya,
  title = {{AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026}},
  author = {Robinson, Nathaniel R. and Abdelmoneim, Shahd and Kantharuban, Anjali and Alsboul, Otba and Lamsiyah, Salima and Marchisio, Kelly and Murray, Kenton},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.1},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.1}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheith A Abandah, Moath R Khaleel, Iyad F Jafar, Mohammad R Abdel-Majeed, Yousef H Hamdan, Ashraf E Suyyagh, Asma A Abdel-Karim, and Shorouq M AlAwawdeh. 2025. Jordanian arabic to modern standard arabic translation using a large model tuned on a purpose-built dataset and synthetic error injection. Jordanian Journal of Computers & Information Technology, 11(3). ↗
  2. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1538–1548. ↗ our card
  3. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. ↗
  4. A. Bergman and Mona Diab. 2022. Towards responsible natural language annotation for the varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022, pages 364–371, Dublin, Ireland. Association for Computational Linguistics. find
  5. Khaleel Hamad and Ahmad Al-Najjar. 2026. A closed-track system for palestinian arabic in the AMIYA shared task. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, Rabat, Morocco. Association for Computational Linguistics. find
  6. Peter J Bickel and Kjell A Doksum. 1977. Mathematical statistics: basic ideas and selected topics. Holden-Day Inc. find
  7. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. ↗
  8. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  10. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  11. Abdelaziz Bounhar. 2025. Atlaset: A moroccan darija text corpus. https://huggingface.co/ datasets/atlasia/Atlaset/. ↗
  12. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. ↗ our card
  13. Shereen Hussein, Mona Farouk, and ElSayed Hemayed. 2019. Gender identification of egyptian dialect in twitter. Egyptian Informatics Journal, 20(2):109– 116. ↗
  14. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics. find
  15. Sujatha Das Gollapalli, Mouad Hakam, Mingzhe Du, and See-Kiong Ng. 2026. NUS-IDS at AMIYA/VarDial 2026: Improving arabic dialectness in llms with reinforcement learning. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, Rabat, Morocco. Association for Computational Linguistics. find
  16. Shankar Kumar and Bill Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176. find our card
  17. Shahd Salah Uddin Dibas, Christian Khairallah, Nizar Habash, Omar Fayez Sadi, Tariq Sairafy, Karmel Sarabta, and Abrar Ardah. 2022. Maknuune: A large open palestinian Arabic lexicon. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 131–141, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  18. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. In Forty-second International Conference on Machine Learning. ↗ our card
  19. Rana Gaber, Yara Allam, Serag Amin, Ranwa Aly, and Bashar Alhafni. 2026. MBZUAI at AMIYA shared task 2026: Adapting open-source llms for dialectal arabic. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, Rabat, Morocco. Association for Computational Linguistics. find
  20. AI Meta. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI Blog. Retrieved December, 20:2024. find
  21. Jonathan Mutal, Perla Al Almaoui, Simon Hengchen, and Pierrette Bouillon. 2026. AladdinFTI @ AMIYA - three wishes for arabic nlp: Fidelity, diglossia, and multidialectal generation. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, Rabat, Morocco. Association for Computational Linguistics. find
  22. Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, and 14 others. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11521– 11567, Bangkok, Thailand. Association for Computational Linguistics. find our card
  23. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. ↗
  24. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. pages 303–324. find
  25. Bashar Talafha, Karima Kadaoui, Samar Mohamed Magdy, Mariem Habiboullah, Chafei Mohamed Chafei, Ahmed Oumar El-Shangiti, Hiba Zayed, Mohamedou Cheikh Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah El Mekki, El Moatez Billah Nagoudi, Benel-hadj Djelloul Mama Saadia, Hamzah A. Alsayadi, Walid Al-Dhabyani, and 8 others. 2024. Casablanca: Data and models for multidialectal Arabic speech recognition. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21745–21758, Miami, Florida, USA. Association for Computational Linguistics. find
  26. Aissam Outchakoucht and Hamza Es-Samaali. 2024. The evolution of darija open dataset: Introducing version 2. Preprint, arXiv:2405.13016. ↗
  27. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  28. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  29. Taghreed Tarmom, William Teahan, Eric Atwell, and Mohammad Ammar Alsalka. 2020. Compression versus traditional machine learning classifiers to detect code-switching in varieties and dialects: Arabic as a case study. Natural Language Engineering, 26(6):663–676. ↗
  30. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. ChatGPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392–418, Singapore. Association for Computational Linguistics. find our card
  31. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  32. Nathaniel Romney Robinson, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. 2025. AL-QASIDA: Analyzing LLM quality and accuracy systematically in dialectal Arabic. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22048–22065, Vienna, Austria. Association for Computational Linguistics. find
  33. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894– 15939, Bangkok, Thailand. Association for Computational Linguistics. ↗ our card
  34. Hashem Sellat, Shadi Saleh, Mateusz Krubiński, Adam Pospíšil, Petr Zemánek, and Pavel Pecina. 2023. UFAL parallel corpus of north levantine 1.0. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL). find
  35. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. ↗
  36. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. A Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Annotator Observations In this section we discuss some of the annotators’ qualitative observations as they conducted the human evaluation. Prompt NUS-IDS Aladdin MBZUAI [..] dataset which contains 2D images, [..] how to find the closest image? Explain in Egyptian Arabic.      KAJ K ½ÊJÓ@ ú G@ ú G@ éʓñÓ  . ,2 XAªK  . @ ú ¯ Pñ“ èY«A ¯ úΫ øñJj  K.  HA .   , èYK  Yg  . èPñ“ ú GY J« ú G@ : èY» àñº Jë  ék. Ag ø ñƒ@ PY  ¯ @ h. A Jm× ú G@  I  J»ð PY ® JK. èPñ“ Ñë @ à@ Qº  ®Jë . , Aî DÓ †Pð P @ AJ KYË@ ú æJ‚ ë Ó   AîE P@ñJ ë ú ÎË@ ð @ , A JÓ@Y ¯ ú Í@ ù ë AëAJ m. ' àñºJë ,Aî DJ K.  ð  A J K.   ¯Q ®Jë  Ó  AîD„jJ ë ú ÎË@ é ®J J ®Ë@ ÐY j J‚ ë ¡J. ¢ËAK.... ? èY» J » ↗ our card

↑ back to top