Elamur Research⌕
☰
home / papers / 2026 / On LLM Prompting Techniques for Arabic Language Arithmetic Reasoning

On LLM Prompting Techniques for Arabic Language Arithmetic Reasoning

Paper · 2026 · Reem Alenezi and Ayed Atallah Salman

Summary

The paper asks how well large language models solve arithmetic word problems written in Arabic, a language with little benchmark data for this task. The authors build three Arabic datasets. AGSM8K is a translation of GSM8K made with ChatGPT and then checked by hand and by a language reviewer. Qudurat comes from Kuwait University aptitude-test items, with a mathematics specialist writing out missing solutions. ArabicMWPs combines translated items from AddSub and MultiArith. On these datasets they test six LLMs, both closed and open, with three prompting strategies: Manual CoT, Zero-shot CoT and Self-consistency. Scoring has two parts. Accuracy is measured on the final answer, and the generated reasoning is compared with reference solutions using BERTScore. The best setup is GPT-4o with Self-consistency, which reaches 97.65% on AGSM8K. The Arabic-specific ALLaM does much worse on the harder data. Fine-tuning on Arabic maths data clearly helps open-source models. Open models also show a large gap between English and Arabic. The work supplies benchmarks and baselines for Arabic mathematical reasoning.

Key points

  • Three new datasets:
    • AGSM8K: 7,473 training and 1,319 test problems needing 2–8 steps.
    • Qudurat: 140 Kuwait University aptitude problems needing 1–8 steps.
    • ArabicMWPs: 725 problems, 265 from AddSub and 460 from MultiArith.
  • GPT-4o with Self-consistency scores 97.65% accuracy on AGSM8K. Its BERTScore on the reasoning text is precision 71.94%, recall 71.31% and F1 71.50%.
  • ALLaM reaches 84.41% on ArabicMWPs but only 43.97% on AGSM8K.
  • Self-consistency was not run for ALLaM because sampling many reasoning paths was too expensive and too slow.
  • Fine-tuning open-source models on Arabic mathematical data improved their results substantially.

Interesting ideas

  • I1. The evaluation scores both the final number and the reasoning text, using BERTScore against worked reference solutions, so a right answer reached by a wrong route can still be detected.
  • I2. Self-consistency samples several independent reasoning paths and keeps the answer that most of them agree on, which reduces one-off logical slips.
  • I3. A benchmark is built by machine-translating an existing one into a JSONL format, followed by manual checks of mathematical consistency and then a separate language reviewer.
  • I4. The three datasets are graded by difficulty, from addition-and-subtraction-only items up to problems needing 8 steps, which shows where a model's ability breaks down; ALLaM's sharp drop between datasets is an example.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Mova's maths word-problem solver could be scored on its explanation trace against gold step lists as well as on the answer, which would separate lucky guesses from sound derivations.
  • I2 (learn): the same majority idea fits silver-data agreement, where several LLM teacher runs or witness parsers vote and only answers with strong agreement become training data.
  • I3 (learn): this layered pipeline (machine draft, logic check, fluency check) could be reused to create or port evaluation sets cheaply while keeping errors under control.
  • I4 (answer): splitting word-problem tests by step count and operation type would show at which reasoning depth Mova's world model stops tracking quantities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kuwait UniversityKuwait City · Kuwait11 works · 147 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSemitic languages · 2,908 works

Cite

Cite as Reem Alenezi and Ayed Atallah Salman. 2026. On LLM Prompting Techniques for Arabic Language Arithmetic Reasoning. The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/52s6fyvjghcs

@inproceedings{alenezi2026llm,
  title = {{On LLM Prompting Techniques for Arabic Language Arithmetic Reasoning}},
  author = {Reem Alenezi and Ayed Atallah Salman},
  booktitle = {The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks},
  year = {2026},
  doi = {10.63317/52s6fyvjghcs},
  url = {https://doi.org/10.63317/52s6fyvjghcs}
}

References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. (2024). Large language models for mathematical reasoning: Progresses and challenges. arXiv:2402.00157. ↗
  2. Albahli, S. (2025). An advanced natural language processing framework for Arabic named entity recognition: A novel approach to handling morphological richness and nested entities. Applied Sciences, 15(6), 3073. ↗
  3. Alghamdi, R., Liang, Z., and Zhang, X. (2022). ArMath: A dataset for solving Arabic math word problems. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), pages 351–362.
  4. Al-Khalifa, S., and Al-Khalifa, H. (2024). The Qiyas benchmark: Measuring ChatGPT mathematical and language understanding in Arabic. arXiv:2407.00146. ↗
  5. Allam, A., Ahmed, S., Hamdi, A., and Mohammed, A. (2025). Arabic large language models for medical text generation. In Proceedings of the 4th International Conference on Computer Technologies (ICCTech) (pp. 1–6).
  6. Amatriain, X. (2024). Prompt design and engineering: Introduction and advanced methods. arXiv:2401.14423. ↗
  7. Bari, M. S., Alnumay, Y., Alzahrani, N. A., Alotaibi, N. M., Alyahya, H. A., AlRashed, S. et al. (2024). Allam: Large language models for Arabic and English. arXiv:2407.15390. ↗
  8. Basem, M., Oshallah, I., Hamdi, A., and Mohamed, A. (2025). Few-shot prompting for extractive Quranic QA with instruction-tuned LLMs. In Proceedings of the Intelligent Methods, Systems, and Applications (IMSA) (pp. 24–29).
  9. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P. et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. ↗
  10. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L. et al. (2021). Training verifiers to solve math word problems. arXiv:2110.14168. ↗
  11. Diao, S., Wang, P., Lin, Y., Pan, R., Liu, X., and Zhang, T. (2023). Active prompting with chain-of-thought for large arXiv:2302.12246. language models. ↗
  12. Hosseini, M. J., Hajishirzi, H., Etzioni, O., and Kushman, N. (2014). Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 523–533).
  13. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv:2205.11916. Kuwait University. Math Test Samples. Available at: https://cem.ku.edu.kw/mathsample ↗
  14. Kwon, S. Y., Bhatia, G., Nagoud, E. M. B., and Abdul-Mageed, M. (2023). ChatGPT for Arabic grammatical error correction. arXiv:2308.04492. ↗
  15. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35. ↗
  16. Lu, P., Qiu, L., Yu, W., Welleck, S., and Chang, K.-W. (2022). A survey of deep learning for mathematical reasoning. arXiv:2212.10535. ↗
  17. Roy, S., and Roth, D. (2016). Solving general arithmetic word problems. arXiv:1608.01413. Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. ↗
  18. K. W., and Lim, E. P. (2023). Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv:2305.04091. ↗
  19. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S. et al. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F. et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903. ↗
  20. World Population Review (2024). Arabic speaking countries. Available at: https://worldpopulationreview.com/country-rankings/arabic-speaking-countries ↗
  21. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 11809–11822. Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y. et al. find
  22. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv:2303.10512. ↗
  23. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). BERTScore: Evaluating text generation with BERT. arXiv:1904.09675. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alenezi-2026-llm-prompting-techniques-arabic-language/