Elamur Research⌕
☰
home / papers / 2025 / GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

Paper · 2025 · Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou et al.

Summary

Open LLMs handle Moroccan Arabic (Darija) poorly. Earlier fixes, such as Atlas-Chat with 458k supervised examples, were costly. This paper asks whether a small, carefully chosen instruction set can bring out Darija ability that a strong multilingual backbone already holds, without hurting its reasoning. The authors machine-translated three curated instruction collections into Darija: LIMA 1K, DEITA 6K and a 44k reasoning-heavy slice of TULU 50K. They kept about a fifth of each set in English and added maths, coding and science prompts. They then trained LoRA adapters on Gemma 3-4B and Gemma 3-27B.

On the 4B model, the DEITA-sized mix raised DarijaMMLU by roughly ten points. The TULU-based mix gave a further gain. At 27B, the resulting GemMaroc-27B performs on par with Atlas-Chat on DarijaMMLU and clearly beats it on DarijaHellaSwag, while keeping strong GSM8K results. The full run cost about 48 GPU·h. The main lesson is that a small amount of well-curated, reasoning-rich data can replace volume when adapting a model to a low-resource dialect.

Key points

  • Training data: Darija translations of LIMA 1K, DEITA 6K and a 44k TULU reasoning slice, each keeping about 20% English originals. All data, code and checkpoints are released.
  • Gemma 3-4B with LoRA (DarijaMMLU): 32.8% for the base model, 34.95% with LIMA, 42.67% with DEITA and 47.53% with TULU. The 1K set was too small to help much.
  • GemMaroc-27B vs Atlas-Chat-27B:
    • DarijaMMLU: 61.6% vs 61.95%.
    • DarijaHellaSwag: 60.5% vs 48.4%.
    • GSM8K: 84.2%.
    • English HellaSwag: 79.4%.
    • Sentiment: about 13.8 points worse.
  • Regression claims vs the table: The text claims no English regression. However, the paper's own table shows the 4B variants falling well below the untuned base on GSM8K (74.75 → 55.95 at best).
  • Compute: 8×H100 for 6 hours, which the authors estimate at about 26 kWh and about 10 kg CO₂e. They put this under 2% of a comparable Atlas-Chat run.
  • Stated limitations: Darija data is machine-translated with little checking, the SentencePiece tokenizer is unchanged, and no large human evaluation has been done yet.

Interesting ideas

  • I1. Mixing a fixed share (about 20%) of untouched source-language examples into translated training data acts as an anchor against forgetting the original skills.
  • I2. A data-size ladder (1K → 6K → 50K) locates the point where adaptation starts to work, instead of assuming that more data is always better.
  • I3. Adding reasoning-heavy examples improved general language benchmarks too, not only maths.
  • I4. Each result is reported next to retention benchmarks for the skills the model should not lose, so a gain on the target task cannot hide damage elsewhere.
  • I5. Translating and reusing existing high-quality curated sets, instead of collecting new data, gives a cheap starting corpus for a new variety.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When retraining the perceptron parser or gparse on silver data from a new domain, always keep a fixed slice of gold treebank sentences in each batch so the old accuracy is protected.
  • I2 (learn): Train components on nested subsets of silver data to find the smallest set that gives the gain, saving LLM-teacher budget.
  • I3 (answer): Mix maths word problems into the training mix for the fable-QA readers and check whether multi-step reasoning carries over between task types.
  • I4 (learn): Pair every new repair rule or retrained model with a fixed battery of untouched tasks, used as negative controls, and reject changes that degrade them.
  • I5 (tree): Convert existing UD treebanks across dialects or domains, and spot-check a small hand-verified sample, since unverified translated data is the weak point this paper itself admits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Hugging FaceNew York · United States14 works · 20,431 citations🎓Université Mohammed VI PolytechniqueMarrakesh · Morocco18 works · 171 citations🎓Institut National des Postes et TélécommunicationsRabat · Morocco2 works · 6 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou, Nouamane Tazi and Ismail Berrada. 2025. GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data. arXiv:2505.17082.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.17082 · arXiv 2505.17082

@misc{skiredj2025gemmaroc,
  title = {{GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data}},
  author = {Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou, Nouamane Tazi and Ismail Berrada},
  year = {2025},
  doi = {10.48550/arxiv.2505.17082},
  eprint = {2505.17082},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.17082}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. International Energy Agency. 2024. Electricity 2025 – Emissions Annex. Average global grid intensity 2024: 0.40 kg CO 2 /kWh. find
  2. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and et al. 2025. Allam: Large language models for arabic and english. In International Conference on Learning Representations. ↗
  3. Abdelaziz Bounhar and Abdeljalil El Majjodi. 2025. Al-atlas: A causal language model for moroccan darija. https://huggingface.co/atlasia/ Al-Atlas-0.5B/. ↗
  4. Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. 2024. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. find
  5. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, Miami, Florida, USA. Association for Computational Linguistics. ↗
  6. Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, and Kai Chen. 2025. Mig: Automatic data selection for instruction tuning by maximizing information gain in semantic space. arXiv preprint arXiv:2504.13835. ↗
  7. Karl Cobbe and et al. 2021. Training verifiers to solve math word problems. In arXiv preprint arXiv:2110.14168. Team Cohere, :, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, Walter Beller-Morales, Alexandre Bérard, Andrew Berneshawi, Anna Bialas, Phil Blunsom, Matt Bobkin, Adi Bongale, Sam Braun, Maxime Brunet, Samuel Cahyawijaya, David Cairuz, Jon Ander Campos, Cassie Cao, Kris Cao, Roman Castagné, Julián Cendrero, Leila Chan Currie, Yash Chandak, Diane Chang, Giannis Chatziveroglou, Hongyu Chen, Claire Cheng, Alexis Chevalier, Justin T. Chiu, Eugene Cho, Eugene Choi, Eujeong Choi, Tim Chung, Volkan Cirik, Ana Cismaru, Pierre Clavier, Henry Conklin, Lucas Crawhall-Stein, Devon Crouse, Andres Felipe Cruz-Salinas, Ben Cyrus, Daniel D’souza, Hugo Dalla-Torre, John Dang, William Darling, Omar Darwiche Domingues, Saurabh Dash, Antoine Debugne, Théo Dehaze, Shaan Desai, Joan Devassy, Rishit Dholakia, Kyle Duffy, Ali Edalati, Ace Eldeib, Abdullah Elkady, Sarah Elsharkawy, Irem Ergün, Beyza Ermis, Marzieh Fadaee, Boyu Fan, Lucas Fayoux, Yannis Flet-Berliac, Nick Frosst, Matthias Gallé, Wojciech Galuba, Utsav Garg, Matthieu Geist, Mohammad Gheshlaghi Azar, Ellen Gilsenan-McMahon, Seraphina Goldfarb-Tarrant, Tomas Goldsack, Aidan Gomez, Victor Machado Gonzaga, Nithya Govindarajan, Manoj Govindassamy, Nathan Grinsztajn, Nikolas Gritsch, Patrick Gu, Shangmin Guo, Kilian Haefeli, Rod Hajjar, Tim Hawes, Jingyi He, Sebastian Hofstätter, Sungjin Hong, Sara Hooker, Tom Hosking, Stephanie Howe, Eric Hu, Renjie Huang, Hemant Jain, Ritika Jain, Nick Jakobi, Madeline Jenkins, JJ Jordan, Dhruti Joshi, Jason Jung, Trushant Kalyanpur, Siddhartha Rao Kamalakara, Julia Kedrzycki, Gokce Keskin, Edward Kim, Joon Kim, Wei-Yin Ko, Tom Kocmi, Michael Kozakov, Wojciech Kryściński, Arnav Kumar Jain, Komal Kumar Teru, Sander Land, Michael Lasby, Olivia Lasche, Justin Lee, Patrick Lewis, Jeffrey Li, Jonathan Li, Hangyu Lin, Acyr Locatelli, Kevin Luong, Raymond Ma, Lukáš Mach, Marina Machado, Joanne Magbitang, Brenda Malacara Lopez, Aryan Mann, Kelly Marchisio, Olivia Markham, Alexandre Matton, Alex McKinney, Dominic McLoughlin, Jozef Mokry, Adrien Morisot, Autumn Moulder, Harry Moynehan, Maximilian Mozes, Vivek Muppalla, Lidiya Murakhovska, Hemangani Nagarajan, Alekhya Nandula, Hisham Nasir, Shauna Nehra, Josh Netto-Rosen, Daniel Ohashi, James Owers-Bardsley, Jason Ozuzu, Dennis Padilla, Gloria Park, Sam Passaglia, Jeremy Pekmez, Laura Penstone, Aleksandra Piktus, Case Ploeg, Andrew Poulton, Youran Qi, Shubha Raghvendra, Miguel Ramos, Ekagra Ranjan, Pierre Richemond, Cécile Robert-Michon, Aurélien Rodriguez, Sudip Roy, Sebastian Ruder, Laura Ruis, Louise Rust, Anubhav Sachan, Alejandro Salamanca, Kailash Karthik Saravanakumar, Isha Satyakam, Alice Schoenauer Sebag, Priyanka Sen, Sholeh Sepehri, Preethi Seshadri, Ye Shen, Tom Sherborne, Sylvie Shang Shi, Sanal Shivaprasad, Vladyslav Shmyhlo, Anirudh Shrinivason, Inna Shteinbuk, Amir Shukayev, Mathieu Simard, Ella Snyder, Ava Spataru, Victoria Spooner, Trisha Starostina, Florian Strub, Yixuan Su, Jimin Sun, Dwarak Talupuru, Eugene Tarassov, Elena Tommasone, Jennifer Tracey, Billy Trend, Evren Tumer, Ahmet Üstün, Bharat Venkitesh, David Venuto, Pat Verga, Maxime Voisin, Alex Wang, Donglu Wang, Shijian Wang, Edmond Wen, Naomi White, Jesse Willman, Marysia Winkels, Chen Xia, Jessica Xie, Minjie Xu, Bowen Yang, Tan Yi-Chern, Ivan Zhang, Zhenyu Zhao, and Zhoujie Zhao. 2025. Command a: An enterprise-ready large language model. ↗
  8. NVIDIA Corporation. 2024. NVIDIA A100 in language models. In Proceedings of the 62nd An-Tensor Core GPU Datasheet. URL: nual Meeting of the Association for Computational https://www.nvidia.com/content/dam/ Linguistics (Volume 1: Long Papers), pages 12012– en-zz/Solutions/Data-Center/a100/pdf/ 12026, Bangkok, Thailand. Association for Compu-nvidia-a100-datasheet-us-nvidia-1758950-r4-web. tational Linguistics. pdf. ↗
  9. Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Devleena Das and Vivek Khetan. 2024. DEFT-UCS: Shaonan Wang, and Chengqing Zong. 2024. X-Data efficient fine-tuning for pre-trained language instruction: Aligning language model in low-models via unsupervised core-set selection for text-resource languages with self-curated cross-lingual editing. In Proceedings of the 2024 Conference on instructions. In Findings of the Association for Com-Empirical Methods in Natural Language Processing, putational Linguistics: ACL 2024, pages 546–566, pages 20296–20312, Miami, Florida, USA. Associa-Bangkok, Thailand. Association for Computational tion for Computational Linguistics. Linguistics. find
  10. Databasemart. 2025. Optimizing vllm perfor-Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, mance on A100 80 gb: Benchmark insights. and Timothy Baldwin. 2023. Bactrian-x: Multilin-https://www.databasemart.com/blog/ gual replicable instruction-following models with vllm-gpu-benchmark-a100-80gb. Accessed low-rank adaptation. May 19, 2025. ↗
  11. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. find
  12. Ember and Energy Institute. 2024. Carbon intensity of Truthfulqa: Measuring how models mimic human electricity generation. https://ourworldindata.falsehoods. In Proceedings of ACL. org/grapher/carbon-intensity-electricity. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Accessed May 19, 2025. ↗
  13. Junxian He. 2024. What makes good data for align-Jindong Han, Hao Liu, Jun Fang, Naiqiang Tan, and Hui ment? a comprehensive study of automatic data se-Xiong. 2025. Automatic Instruction Data Selection lection in instruction tuning. In The Twelfth Interna-for Large Language Models via Uncertainty-Aware tional Conference on Learning Representations. Influence Maximization. In Proceedings of the ACM Yuren Mao, Yaobo Liang, Nan Duan, Haobo Wang, Web Conference 2025, pages 4969–4979, Sydney, Kai Wang, Lu Chen, and Yunjun Gao. 2022. Less-Australia. ACM. forgetting multi-lingual fine-tuning. In Advances in Peter Henderson, Jieru Hu, Joshua Romoff, Emma Neural Information Processing Systems 35: NeurIPS Brunskill, Dan Jurafsky, and Joelle Pineau. 2020. 2022, New Orleans, USA. Towards the systematic reporting of the energy Ministry of Digital Transition and Adminis-and carbon footprints of machine learning. CoRR, tration Reform. 2024. Digital Morocco abs/2002.05651. Revised version published 2022. 2030: National Strategy. https://www.mmsp. Dan Hendrycks, Collin Burns, Steven Basart, Andy gov.ma/sites/default/files/2024-09/ Zou, Mantas Mazeika, Dawn Song, and Jacob Stein-PlaquetteInstitutionnel_18092024_Ang.pdf. hardt. 2021. Measuring massive multitask language Accessed: 2025-05-13. understanding. In Proceedings of the International NVIDIA. 2024. NVIDIA A100 Tensor Core GPU Conference on Learning Representations (ICLR). datasheet. https://www.nvidia.com/content/ ↗
  14. Oskar Holmström and Ehsan Doostmohammadi. 2023. dam/.../a100-80gb-datasheet.pdf. Accessed Making instruction finetuning accessible to non-May 19, 2025. English languages: A case study on Swedish models. find
  15. ProtectAI. 2025. protectai/xlm-roberta-In Proceedings of the 24th Nordic Conference on base-language-detection-onnx. https: Computational Linguistics (NoDaLiDa), pages 634– //huggingface.co/protectai/ 642, Tórshavn, Faroe Islands. University of Tartu xlm-roberta-base-language-detection-onnx. Library. Accessed May 2025. ONNX version of papluca/xlm-Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-roberta-base-language-detection using the Optimum Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu library. Fine-tuned on the Language Identification Chen. 2022. LoRA: Low-rank adaptation of large dataset. language models. In International Conference on Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Learning Representations. Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ah-Imane Momayiz, Xuguang Ren, Eric Moulines, met Üstün, Anna Korhonen, and Hinrich Schütze. find
  16. Preslav Nakov, Michalis Vazirgiannis, and Eric Xing. 2024. Muri: High-quality instruction tuning datasets 2025. Atlas-chat: Adapting large language models for low-resource languages via reverse instructions. for low-resource moroccan arabic dialect. In Proceedings of the First Workshop on Language Mod-Alexandre Lacoste, Alexandra Luccioni, Victor els for Low-Resource Languages, pages 9–30, Abu Schmidt, and Thomas Dandres. 2019. Quantifying Dhabi, United Arab Emirates. Association for Com-the carbon emissions of machine learning. arXiv putational Linguistics. preprint arXiv:1910.09700. ↗
  17. Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multi-Emma Strubell, Ananya Ganesh, and Andrew McCal-lingual instruction tuning for reasoning consistency lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  18. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An arabic-centric multimodal generative ai platform. ↗
  19. Alexander Arno Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, and Mehdi Ali. 2024. Investigating multilingual instruction-tuning: Do polyglot models demand for multilingual instructions? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20829–20855, Miami, Florida, USA. Association for Computational Linguistics. ↗
  20. Junhong Wu, Yang Zhao, Yangyifan Xu, Bing Liu, and Chengqing Zong. 2025. Boosting LLM translation skills without general ability loss via rationale distillation. In Submitted to ACL Rolling Review - February 2025. Under review. find
  21. Minghao Wu, Thuy-Trang Vu, Lizhen Qu, and Gholamreza Haffari. 2024. MIXTURE-OF-SKILLS: Learning to optimize data usage for fine-tuning large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14226–14240. Association for Computational Linguistics. find
  22. Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. 2024. Rethinking Data Selection at Scale: Random Selection is Almost All You Need. arXiv preprint arXiv:2410.09335. ↗
  23. Silvana Yakhni and Ali Chehab. 2025. Fine-tuning llms for low-resource dialect translation: The case of lebanese. ↗
  24. Xiao Yu, Qingyang Wu, Yu Li, and Zhou Yu. 2024. LIONs: An empirically optimized approach to align language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8732–8753, Miami, Florida, USA. Association for Computational Linguistics. find
  25. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL. ↗
  26. Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2024. PLUG: Leveraging pivot language in cross-lingual instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7025–7046, Bangkok, Thailand. Association for Computational Linguistics. ↗
  27. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. In Advances in Neural Information Processing Systems (NeurIPS-2023), volume 36, pages 55006–55021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/skiredj-2025-gemmaroc-unlocking-darija-proficiency-llms/