Elamur Research⌕
☰
home / papers / 2026 / State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation

Paper · 2026 · Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair et al.

Summary

Arabic is still poorly served by large language models. Proprietary systems cannot be retrained locally, and Arabic-first models trained from scratch are expensive and still weak at reasoning. The authors adapt the open reasoning model DeepSeek-R1, which has a sparse mixture-of-experts backbone, using parameter-efficient fine-tuning instead of new pretraining. Their supervision set has 372M tokens and is filtered for benchmark contamination. About 80% of it is Arabic (Modern Standard Arabic plus Gulf, Levantine and Egyptian dialects, preferring natively written text). The remaining 20% is English, kept as an anchor against forgetting. The set is spread across several genres. Training uses a four-phase chain-of-thought distillation that adds explicit Arabic linguistic checks and regional ethical norms. The resulting Arabic-DeepSeek-R1 has the highest average on the seven-task Open Arabic LLM Leaderboard (OALL v2), ahead of the best open model and GPT-5.1. The gains are largest on grammar and safety tasks. The authors conclude that Arabic's lag comes mainly from too little specialization, not from model architecture. Adapting a strong open backbone is therefore a cheap, locally controllable route for low-resource languages.

Key points

  • OALL average of 80.18%:
    • +4.32 over D2IL-Arabic-Qwen2.5-72B
    • +2.31 over GPT-5.1
    • +5.28 over Falcon-H1-Arabic-34B-Instruct
    • +14.75 over Jais-family-30B-16k-chat
  • The model is SOTA or near-SOTA on 5 of 7 benchmarks. The largest margin is on MadinahQA (+8.43 over the category leader), with strong results on AraTrust, AlGhafa and ALRAGE.
  • Composition of the 372M-token supervision set:
    • 103.2M literature/critical analysis
    • 90M STEM/math/logic
    • 70M creative writing/dialogue
    • 60.2M reviews/service feedback
    • 40M legal/cultural alignment
    • 8.6M social/dialectal
  • The evaluation suite also includes ArabicMMLU, Arabic EXAMS and ArbMMLU-HT. On EXAMS, Llama-3.3-70B (66.67%) still leads.
  • On ALRAGE the model is only +0.16 over its baseline, though it beats the category leader by 5.84 and GPT-5.1 by 4.52. The authors explain this by their CoT data targeting reasoning rather than retrieval.

Interesting ideas

  • I1. Keeping a fixed minority share of the original-distribution data during adaptation helps the model keep its base abilities while it specializes.
  • I2. Teacher reasoning traces are filtered through explicit language-specific checks before being used as supervision, rather than trusting the teacher's output as is.
  • I3. The training set is planned as an explicit token budget per genre, which makes the mixture auditable and easy to rebalance.
  • I4. Natively written text is preferred over machine-translated text, because translations carry artefacts of the source language.
  • I5. Results are read per benchmark, not only as an average, which showed that reasoning-focused supervision did little for retrieval tasks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When retraining gparse on new domains or converted UD dialects, reserve a constant slice of the original treebank in every batch so that LAS on the old data does not drift.
  • I2 (learn): Answers from the LLM teacher tiers could pass morphology and agreement checks from crate en before entering silver data, and rejected cases could feed the repair rules.
  • I3 (tree): Build silver treebanks with stated per-genre quotas (fiction, fables, school texts, dialogue) and report parser accuracy per quota.
  • I4 (tree): When collecting English text for witness-parser agreement, prefer original English sources and keep translated text apart, so translation syntax does not leak into the silver trees.
  • I5 (answer): Report school tests, maths word problems and fable QA separately to see which kind of supervision helps which reader, and target the weakest category next.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSemitic languages · 2,908 worksEfficient models & compression · 242 works

Cite

Cite as Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair, Jyani Akshay Jagdishbhai, Atul Yaduvanshi, Amarendra Chaudhary, Madalina Ciobanu, Qingqing Mao and Ritankar Das. 2026. State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation. arXiv:2604.06421.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.06421 · arXiv 2604.06421

@misc{singh2026state,
  title = {{State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation}},
  author = {Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair, Jyani Akshay Jagdishbhai, Atul Yaduvanshi, Amarendra Chaudhary, Madalina Ciobanu, Qingqing Mao and Ritankar Das},
  year = {2026},
  doi = {10.48550/arxiv.2604.06421},
  eprint = {2604.06421},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.06421}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abboud, K., Golovneva, O., and DiPersio, C. Cross-lingual transfer for low-resource Arabic language understanding. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), 2022.
  2. Our results point to several directions for future work. First, integrating retrieval-aware training objectives or including RAG-style supervision during fine-tuning may help recover and surpass baseline performance on ALRAGE. Second, lightweight domain-adaptive phases targeted at exam curricula and RAG corpora could complement the current general-Al-Khalifa, S., Durrani, N., Al-Khalifa, H., and Alam, F. The landscape of Arabic large language models. Communications of the ACM, 68(10):54–61, 2025. find
  3. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., . . . , and Zhang, Z. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025. find
  4. Alghamdi, E. A., Masoud, R. I., Alnuhait, D., Alomairi, A. Y., Ashraf, A., and Zaytoon, M. Aratrust: An evaluation of trustworthiness for LLMs in Arabic. arXiv preprint arXiv:2403.09017, 2024. ↗
  5. Aljahdali, L. and Kaki, J. Exploring Arabic large language models: A comprehensive review. Effat Undergraduate Research Journal, 4:1–11, 2023. find
  6. Hardalov, M., Mihaylov, T., Zlatkova, D., Dinkov, Y., Koychev, I., and Nakov, P. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of EMNLP 2020, 2020. find
  7. AlKhamissi, B., ElNokrashy, M., AlKhamissi, M., and Diab, M. Investigating cultural alignment of large language models. arXiv preprint arXiv:2402.13231, 2024. ↗
  8. Koloski, B., Škrlj, B., Robnik-Šikonja, M., and Pollak, S. Measuring catastrophic forgetting in cross-lingual classification. IEEE Access, 13:33509–33520, 2025. find
  9. Almazrouei, E., Cojocaru, R., Baldo, M., Malartic, Q., Alobeidli, H., Mazzotta, D., Penedo, G., Campesan, G., Farooq, M., Alhammadi, M., Launay, J., and Noune, B. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, 2023.
  10. Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A., Alraeesi, A., Almubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., Habash, N., Nakov, P., and Baldwin, T. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of ACL 2024, 2024.
  11. Alzubaidi, A., Alsuwaidi, S., Boussaha, B. E. A., AlQadi, L., Alkaabi, O., Alyafeai, M., Alobeidli, H., and Hacid, H. Evaluating Arabic large language models: A survey of benchmarks, methods, and gaps. arXiv preprint arXiv:2510.13430, 2025. ↗
  12. Mashaabi, M., Al-Khalifa, S., and Al-Khalifa, H. A survey of large language models for Arabic language and its dialects. arXiv preprint arXiv:2410.20238, 2024. ↗
  13. Bari, M. S., Alnumay, Y., Alzahrani, N. A., Alotaibi, N. M., Alyahya, H. A., AlRashed, S., Mirza, F. A., Alsubaie, S. Z., Alahmed, H. A., Alabduljabbar, G., Alkhathran, R., Almushayqih, Y., Alnajim, R., Alsubaihi, S., Mansour, M. A., Alrubaian, M., Alammari, A., Alawami, Z., Al-Thubaity, A., Abdelali, A., Kuriakose, J., Abujabal, A., Al-Twairesh, N., Alowisheq, A., and Khan, H. Allam: Large language models for Arabic and English. arXiv preprint arXiv:2407.15390, 2024. ↗
  14. OpenAI. GPT-5.1: A smarter, more conversational Chat-GPT, 2025. URL https://openai.com/index/ gpt-5-1/. ↗
  15. Sengupta, N., Sahu, S. K., Jia, B., Katipomu, S., Li, H., Koto, F., Marshall, W., Gosal, G., Liu, C., Chen, Z., Afzal, O. M., Kamboj, S., Pandit, O., Pal, R., Pradhan, L., Mujahid, Z. M., Baali, M., Han, X., Bsharat, S. M., ..., and Xing, E. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149, 2023. ↗
  16. Common Crawl. Statistics of common crawl monthly archives: Distribution of languages. https://commoncrawl.github.io/ cc-crawl-statistics/plots/languages, 2025. Accessed: 2026-01-09. ↗
  17. Technology Innovation Institute. Open Arabic LLM leaderboard, 2024. URL https://huggingface.co/spaces/OALL/ Open-Arabic-LLM-Leaderboard. ↗
  18. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. ↗
  19. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. ↗
  20. El Filali, A., Aloui, M., Husaain, T., Alzubaidi, A., Boussaha, B. E. A., Cojocaru, R., Fourrier, C., Habib, N., and Hacid, H. The Open Arabic LLM leaderboard 2. Hugging Face Blog, 2025. URL https://huggingface.co/blog/leaderboard-arabic-v2. ↗
  21. Zhang, D., Song, J., Bi, Z., Song, X., Yuan, Y., Wang, T., Yeong, J., and Hao, J. Mixture of experts in large language models. arXiv preprint arXiv:2507.11181, 2025. ↗
  22. Fireworks AI. DeepSeek v3 and R1 model architecture: Why it’s powerful and economical, 2025. URL https://fireworks.ai/blog/ deepseek-model-architecture. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-state-art-arabic-language-modeling/