Elamur Research⌕
☰
home / papers / 2025 / Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

Paper · 2025 · Guokan Shang, Hadi Abdine, Yousef Khoubrane et al.

Summary

Arabic-focused LLMs such as Jais, AceGPT and ALLaM mostly balance English and Modern Standard Arabic and largely ignore spoken dialects. This work targets Moroccan Arabic (Darija), which about 40 million people speak but which has little data, no fixed orthography, and appears in both Arabic and Latin script (Arabizi). Pretraining was not feasible with so little text, so the authors instruction-tuned Gemma 2 base models at 2B, 9B and 27B to produce Atlas-Chat. They also skipped continual pretraining. The training set, Darija-SFT-Mixture, has 458K samples drawn from three sources: curated native Darija datasets, instructions synthesized by proprietary models, and English instruction sets translated under quality checks. A new evaluation suite covers knowledge, instruction following and classic NLP tasks: DarijaMMLU, DarijaHellaSwag, DarijaAlpacaEval and DarijaBench. Atlas-Chat beats general and Arabic-specialized models, including larger ones. All resources are public, which gives a reusable recipe for adapting LLMs to low-resource language varieties.

Key points

  • Darija-SFT-Mixture contains 458K instructions. Ten native datasets cover translation, summarization and sentiment. Template conversion produced 80% zero-shot, 10% few-shot and 10% multi-turn samples.

  • DODa-10K takes the first 10K Darija–English pairs from the DODa corpus and adds MSA and French via GPT-4. The result is quintuples (Darija in Arabic and Latin script, MSA, English, French), reviewed by native speakers. Translation instructions cover six directions and are always phrased in Darija.

  • The 9B model scores 13% higher than a 13B competitor on DarijaMMLU.

  • Atlas-Chat-27B leads the best competitor zero-shot on four benchmarks:

    BenchmarkLead
    DarijaMMLU10.07%
    DarijaHellaSwag12.76%
    Belebele_Ary1.51%
    DarijaAlpacaEval10.00%

  • On summarization, an LLM judge preferred Atlas-Chat-27B output over the reference summaries 60.70% of the time. That is roughly 45 points above Llama-3.1-8B-Instruct.

  • Baselines scored very low BLEU because they produced MSA or mixed MSA/Darija instead of Darija.

  • Stated limitations:

    • hallucinations
    • possible dataset bias
    • heavy reliance on Claude for translating instructions
    • no preference tuning

Interesting ideas

  • I1. Native speakers set a crisp admission rule for data: Darija mixed with MSA is acceptable, but mixing with other dialects disqualifies a sample. This keeps a corpus coherent without demanding purity.
  • I2. Without data for continual pretraining, they built synthetic instructions that resemble next-word prediction over longer passages. This injects raw-language exposure through the fine-tuning channel.
  • I3. Before trusting BLEU, they checked what the low-scoring outputs actually were. Baselines were answering in the wrong language variety, so n-gram overlap collapsed for reasons unrelated to translation quality.
  • I4. Where ROUGE could not capture valid variation in summaries, they had an LLM judge compare system output against the reference itself.
  • I5. Mixing prompt formats (mostly zero-shot, some few-shot and multi-turn) in fixed proportions trains one model to handle several interaction styles.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When converting or merging treebanks across UD dialects, a simple written rule about which mixtures are tolerated could keep training sets consistent.
  • I2 (learn): Silver training data could include continuation-style tasks over long texts, giving the LLM teacher tiers a cheap way to transfer fluency in thin domains.
  • I3 (learn): Evaluation could add a negative-control check that output has the right register or variety before scoring, so metric failures are diagnosed rather than misread.
  • I4 (back): Text generated from the meaning graph could be judged pairwise against the original text instead of only by n-gram overlap, since equally valid paraphrases are expected.
  • I5 (answer): School-test and fable-QA training items could be rendered in several formats in set ratios, so the readers do not overfit to a single question layout.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École PolytechniquePalaiseau · France15 works · 6,688 citations🎓KTH Royal Institute of TechnologyStockholm · Sweden20 works · 1,723 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksLow-resource languages · 287 works

Cite

Cite as Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis and Eric Xing. 2025. Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect. Proceedings of the First Workshop on Language Models for Low-Resource Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shang2025atlas,
  title = {{Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect}},
  author = {Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis and Eric Xing},
  booktitle = {Proceedings of the First Workshop on Language Models for Low-Resource Languages},
  year = {2025},
  url = {https://aclanthology.org/2025.loreslm-1.2/}
}

References

50 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. QADI: Arabic dialect identification in the wild. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  3. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.
  4. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  6. Abdeljalil Elouardighi, Mohcine Maghfour, and Haf-dalla Hammia. 2017. Collecting and processing arabic facebook comments for sentiment analysis. In Model and Data Engineering: 7th International Conference, MEDI 2017, Barcelona, Spain, October 4–6, 2017, Proceedings 7, pages 262–274. Springer. find
  7. Kabil Essefar, Hassan Ait Baha, Abdelkader El Mahdaouy, Abdellah El Mekki, and Ismail Berrada. 2023. Omcd: Offensive moroccan comments dataset. Language Resources and Evaluation, 57(4):1745–1765. ↗
  8. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. ↗
  9. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135. ↗
  10. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2022. Automatic text summarization for moroccan arabic dialect using an artificial intelligence approach. In International Conference on Business Intelligence, pages 158–177. Springer. find
  11. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2024. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, pages 1–13. de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  12. Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, and Firoj Alam. 2024. Native vs non-native language prompting: A comparative analysis. arXiv preprint arXiv:2409.07054. ↗
  13. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2024. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36. ↗
  14. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics ACL 2024, pages 5622–5640, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. ↗
  15. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. ↗
  16. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  17. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  18. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PMLR. ↗
  19. Jérôme Louradour, Julie Hunter, Ismaïl Harrando, Guokan Shang, Virgile Rennard, and Jean-Pierre Lorré. 2024. Claire: Large language models for spontaneous French dialogue. In Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position, pages 530–548, Toulouse, France. ATALA and AFPC. find
  20. Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzman. 2023. Small data, big impact: Leveraging minimal data for effective machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation. find
  21. Moncef Garouani and Jamal Kharroubi. 2021. Mac: an open and free moroccan arabic corpus for sentiment analysis. In The Proceedings of the International Conference on Smart City Applications, pages 849– 858. Springer. find
  22. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644. ↗
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  24. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  25. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  26. Julie Hunter, Jérôme Louradour, Virgile Rennard, Ismaïl Harrando, Guokan Shang, and Jean-Pierre Lorré. 2023. The claire french dialogue dataset. arXiv preprint arXiv:2311.16840. ↗
  27. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702. ↗
  28. Mouad Jbel, Mourad Jabrane, Imad Hafidi, and Abdul-mutallib Metrane. 2024. Sentiment analysis dataset in moroccan dialect: bridging the gap between arabic and latin scripted dialect. Language Resources and Evaluation, pages 1–30. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego 1: Long Papers), pages 2740–2756, Toronto, Canada. Association for Computational Linguistics. ↗
  29. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707. ↗
  30. Ahmed Oussous, Fatima-Zahra Benjelloun, Ayoub Ait Lahcen, and Samir Belfkih. 2020. Asa: A framework for arabic sentiment analysis. Journal of Information Science, 46(4):544–559. ↗
  31. Ahmed Oussous, Ayoub Ait Lahcen, and Samir Belfkih. 2018. Improving sentiment analysis of moroccan tweets using ensemble learning. In Big Data, Cloud and Applications: Third International Conference, BDCA 2018, Kenitra, Morocco, April 4–5, 2018, Revised Selected Papers 3, pages 91–104. Springer. find
  32. Aissam Outchakoucht and Hamza Es-Samaali. 2021. Moroccan dialect-darija-open dataset. arXiv preprint arXiv:2103.09687. ↗
  33. Aissam Outchakoucht and Hamza Es-Samaali. 2024. The evolution of darija open dataset: Introducing version 2. arXiv preprint arXiv:2405.13016. ↗
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  35. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. ↗
  36. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗
  37. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  38. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. find
  39. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. ↗
  40. Atnafu Lambebo Tonja, Bonaventure FP Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, et al. 2024. Inkubalm: A small language model for low-resource african languages. arXiv preprint arXiv:2408.17024. ↗
  41. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  42. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics. find
  43. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. ↗
  44. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244. ↗
  45. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  46. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. ↗
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  48. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815. ↗
  49. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  50. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shang-2025-atlas-chat-adapting-large-language/