Elamur Research⌕
☰
home / papers / 2025 / AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP

AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP

Paper · 2025 · Ahmed Abul Hasanaath, Aisha Alansari, Ahmed Ashraf et al.

Summary

AraReasoner asks how well recent reasoning-oriented LLMs handle Arabic. Arabic is hard for models because of its rich morphology, many dialects, and non-Latin script, and earlier Arabic benchmarks mostly covered BERT-style encoders or only a few closed GPT models. The authors assembled fifteen public Arabic datasets in three groups: sentence classification (8 tasks), generation (5 tasks), and linguistic analysis (WSD and PoS). They tested DeepSeek R1 and V3, distilled R1-Qwen variants, QwQ-32B, GPT-4o and o4-mini. Each model was evaluated zero-shot, with 3 or 5 in-context examples, and, for small R1-Qwen models, after LoRA fine-tuning. Before the main runs, a prompt-style selection step chose a template for each task. Three in-context examples raise classification F1 sharply, while five examples sometimes help less. DeepSeek reasoning models beat o4-mini zero-shot on inference-heavy tasks. LoRA tuning of small models can beat simply using a bigger model. The study gives practitioners a map of which adaptation lever to use for a morphologically rich language.

Key points

  • Fifteen tasks drawn from AJGT, Ar-Sarcasm, the Mubarak et al. hate/offensive set, Khouja's claim/stance data, Seelawi et al. paraphrase pairs, AraSum→EASC (summarization), APPSD→APB, BOLT, UNv1, XTREME, Universal Dependencies v2.3 (PoS) and an Arabic WSD benchmark. Metrics are F1, BLEU, ROUGE and SQuAD F1.
  • Three in-context examples add over 13 F1 points on average for classification. Sentiment jumps to 87.5% (previously 35.3%), and paraphrase detection reaches 87.0% (previously 56.1%). Going to five examples lowered scores on StD and DD.
  • Zero-shot, DeepSeek reasoning models lead o4-mini by about 12 F1 on complex inference. R1-671B averages 25.17 points above its 14B variant. QwQ-32B trails R1-Q32B by 10.62 points despite a similar parameter count.
  • R1-671B is stronger on classification and linguistic analysis, except sarcasm detection. GPT-4o wins most generation tasks.
  • Zero-shot PoS scores are near 5 for every model. After LoRA fine-tuning, R1-Q14B reaches 95.99 on PoS, 86.27 on WSD and an 83.85 classification average.

Interesting ideas

  • I1. The summarizer is fine-tuned on one corpus (AraSum) but tested on a different one (EASC), so the score reflects transfer rather than memorizing a single corpus's style.
  • I2. Before benchmarking, each task gets its own prompt, chosen by comparing instructive, interrogative and role-playing phrasings of the same instruction. This separates the effect of wording from the effect of the model.
  • I3. Scores do not always rise with more examples: going from three to five examples hurt some tasks, which suggests that extra demonstrations can introduce conflicting patterns.
  • I4. Zero-shot PoS was close to zero while fine-tuned PoS was near 96, so token-level structured output needs adaptation that semantic classification does not.
  • I5. One annotated corpus can provide two label layers, as Ar-Sarcasm does for both sarcasm and dialect, giving two tasks from a single collection.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could train parsers or frame extractors on one source and evaluate on another treebank or genre, so that the measured gains are not artifacts of the corpus.
  • I2 (learn): Mova's LLM teacher tiers could compare several phrasings per labeling task and keep the most reliable one before producing silver data.
  • I3 (learn): when Mova prompts a teacher LLM, it should sweep the number of examples as a tuned parameter and not assume that more examples help.
  • I4 (tokens): Mova should not trust raw LLM tags as tagger supervision; a small tuned model, or tags checked against the tagger, is the safer source.
  • I5 (frames): Mova could reuse a single annotated text set for several pragmatic labels (speech act, irony), so errors on one layer can be compared against another on the same sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia52 works · 4,666 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSemitic languages · 2,908 works

Cite

Cite as Ahmed Abul Hasanaath, Aisha Alansari, Ahmed Ashraf, Salmane Chafik, Hamzah Luqman and Saad Ezzini. 2025. AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.1028

@inproceedings{hasanaath2025arareasoner,
  title = {{AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP}},
  author = {Ahmed Abul Hasanaath, Aisha Alansari, Ahmed Ashraf, Salmane Chafik, Hamzah Luqman and Saad Ezzini},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.1028},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.1028}
}

References

37 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023. Larabench: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982. ond International Conference on Data Science, E-Learning and Information Systems, pages 1–5. ↗
  2. Badr AlKhamissi, Muhammad N ElNokrashy, and Mohamed Gabr. 2020. Deep diacritization: Efficient hierarchical recurrence for improved arabic diacritization. arXiv preprint arXiv:2011.00538. ↗
  3. Khaled Mohammad Alomari, Hatem M ElSherif, and Khaled Shaalan. 2017. Arabic tweets sentimental analysis using machine learning. In International conference on industrial, engineering and other applications of applied intelligent systems, pages 602– 610. Springer. find
  4. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184.
  5. Z Alyafeai, MS Alshaibani, B AlKhamissi, H Luqman, E Alareqi, and A Fadel. 2023a. Taqyim: Evaluating arabic nlp tasks using chatgpt models. arxiv 2023. arXiv preprint arXiv:2306.16322. ↗
  6. Zaid Alyafeai, Maged S Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023b. Taqyim: Evaluating arabic nlp tasks using chatgpt models. arXiv preprint arXiv:2306.16322. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  8. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗
  9. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. ↗
  10. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45. ↗
  11. Noora Al-Shameri and Hend Al-Khalifa. 2024. Arabic paraphrased parallel synthetic dataset. Data in Brief, 57:111004. find
  12. Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint arXiv:2502.15652. ↗
  13. Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building arabic paraphrasing benchmark. In Proceedings of the Sec-Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox. 2010. Using mechanical turk to create a corpus of arabic summaries. find
  14. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6):2567. ↗
  15. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758. ↗
  16. Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In The 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 32–39. European Language Resources Association (ELRA). with a shared task on offensive language detection, pages 48–52.
  17. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  18. Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. 2022. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1):89–122. ↗
  19. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025a. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. ↗
  20. Joakim Nivre, Mitchell Abrams, Željko Agić, and Bengt Ahrenberg. 2018. Universal dependencies 2.3. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  21. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025b. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. ↗
  22. Qwen Qwen-Team. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  23. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354. ↗
  24. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T Al-Natsheh. 2019. Nsurl-2019 shared task 8: Semantic question similarity in arabic. arXiv preprint arXiv:1909.09691. ↗
  25. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  26. Mram Kahla, Zijian Győző Yang, and Attila Novák. 2021. Cross-lingual fine-tuning for abstractive Arabic text summarization. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 655–663, Held Online. INCOMA Ltd.
  27. Jude Khouja. 2020. Stance prediction and claim verification: An arabic perspective. In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 8–17.
  28. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. ↗
  29. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024b. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. ↗
  30. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
  31. Ali Shazal, Aiza Usman, and Nizar Habash. 2020a. A unified model for arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the fifth arabic natural language processing workshop, pages 167–177.
  32. Ali Shazal, Aiza Usman, and Nizar Habash. 2020b. A unified model for arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the fifth arabic natural language processing workshop, pages 167–177.
  33. Aditya Siddhant, Junjie Hu, Melvin Johnson, Orhan Firat, and Sebastian Ruder. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of the International Conference on Machine Learning 2020, pages 4411–4421. ↗
  34. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  36. Jingbo Zhu, Huizhen Wang, Benjamin K Tsou, and Matthew Ma. 2009. Active learning with sampling by uncertainty and density for data annotations. IEEE Transactions on audio, speech, and language processing, 18(6):1323–1331. ↗
  37. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hasanaath-2025-arareasoner-evaluating-reasoning-based-llms/