Elamur Research⌕
☰
home / papers / 2026 / SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning

SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning

Paper · 2026 · Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl et al.

Summary

Arabic financial NLP has almost no evaluation resources, even though Gulf economies and the Islamic finance industry produce large amounts of Arabic regulatory, juristic and corporate text. Islamic finance also requires Shari'ah compliance checks, which current benchmarks ignore. SAHM fills this gap with a single benchmark of seven tasks:

  • questions on AAOIFI standards,
  • fatwa-based QA and MCQ,
  • accounting and business exam questions,
  • sentiment classification of financial reports,
  • extractive summarization,
  • event-cause reasoning.

The items come from genuine standards, fatwa archives from seven countries and company disclosures. They were built with an OCR plus Gemini-assisted generation pipeline and checked by humans. The authors test 20 LLMs. Models that do very well when picking among options do much worse when they must write an answer, and Arabic-specific base models land near the bottom of the ranking. Fine-tuning small Arabic models on SAHM closes much of this gap: 7–8B models overtake GPT-5 on some exams and match 72B open models on average. The work shows that domain adaptation, not fluency or size alone, drives trustworthy financial reasoning.

Key points

  • SAHM contains 14,380 expert-verified instances across 7 tasks, drawn from AAOIFI standards, fatwas, accounting/business exams and corporate reports.
  • Across 20 LLMs, scores reach about 91% on recognition-style tasks but fall sharply on generation. Event-cause QA shows the widest spread (1.89–9.84 out of 10).
  • Fine-tuning raises Accounting scores by as much as +26 points and Business scores by as much as +25.
    • SAHM-ALLAM-7B reaches 93.99% on Business MCQ versus 72.68% for GPT-5.
    • SAHM-JAIS-8B improves on every task.
    • Both roughly match 72B baselines, about 10× fewer parameters.
  • SILMA-9B serves as a contrast case: fine-tuning it gave mixed results, showing that the outcome of adaptation depends on the base model.
  • On fatwa rulings, accuracy rises from about 28% when the reference answer cites no Qur'an/hadith evidence to about 55% when it cites six or more. The authors note this may reflect question easiness rather than true grounding.

Interesting ideas

  • I1. Testing the same domain knowledge in both a choose-from-options format and a free-answer format exposes models that can recognize answers but cannot produce them.
  • I2. When a correlation appears (more cited evidence, higher accuracy), the authors explicitly state both a hopeful and a skeptical explanation instead of claiming grounding.
  • I3. The authors argue that scoring should check whether the cited sources and sections are correct, not just whether the final answer is right.
  • I4. Shari'ah items are judged against the provided sources under stated assumptions, not by settling which school of thought is correct.
  • I5. The authors flag mixed numeral systems and embedded foreign acronyms as a specific source of difficulty in financial text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course readers could be scored on paired MCQ and open versions of each question, revealing where the correct option is picked without a real derivation in the world model.
  • I2 (learn): Every apparent gain in evaluation could get a matched negative control, such as questions with equal difficulty but no supporting facts, to separate real use of evidence from easy items.
  • I3 (answer): Since Mova's steps are explainable, the benchmark could also check that each answer points to the correct tree nodes and world facts, penalizing right answers backed by wrong justifications.
  • I4 (world): The world of a text should answer relative to what that text asserts, kept separate from compiled background knowledge, so fables and counterfactual stories are judged on their own terms.
  • I5 (tokens): The en tokenizer could normalize varied number formats (digits, words, currency, percentages) and keep acronyms as single lemmas, so that later measure and value extraction stays reliable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria21 works · 2,595 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov and Zhuohan Xie. 2026. SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1593

@inproceedings{elbadry2026sahm,
  title = {{SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning}},
  author = {Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov and Zhuohan Xie},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1593},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1593}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur A. Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed K. Elmagarmid, Mohamed Y. Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An Arabic-centric multimodal generative AI platform. ArXiv preprint, abs/2501.13944. ↗
  2. Hayfa A Aleid and Aqil M Azmi. 2025. Hajj-FQA: A benchmark Arabic dataset for developing question-answering systems on Hajj fatwas. Journal of King Saud University Computer and Information Sciences, 37(6):135. ↗
  3. Salem Alhajraf. 2025. Strategic role of sovereign wealth funds in the Gulf’s energy transition and economic diversification. Technical report, Rice University’s Baker Institute for Public Policy. ↗
  4. Ohoud Alyemny, Hend S. Al-Khalifa, and Abdulrahman A. Mirza. 2023. A data-driven exploration of a new Islamic fatwas dataset for Arabic NLP tasks. Data, 8(10):155. ↗
  5. Anthropic. 2025a. System Card: Claude Haiku 4.5. Anthropic. find
  6. Anthropic. 2025b. System Card: Claude Opus 4.5. Anthropic. Ethical Statement and Broad Impact Licensing. We release SAHM under a dual license: (1) code and evaluation scripts under MIT License, and (2) annotation data under CC BY-NC 4.0, restricting commercial use while enabling academic research. Users must independently obtain source documents where applicable. Availability. The benchmark data, models, and evaluation scripts are publicly available at https://huggingface.co/SahmBenchmark, https://github.com/mbzuai-nlp/SAHM. ↗
  7. Anthropic. 2025c. System Card: Claude Sonnet 4.5. Anthropic. find
  8. Dogu Araci. 2019. FinBERT: Financial sentiment analysis with pre-trained language models. ArXiv preprint, abs/1908.10063. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Ma-jed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuri-akose, Abdalghani Abujabal, Nora Al-Twairesh, dataset in modern and dialectical Arabic. In Proceedings of ArabicNLP 2023, pages 276–287, Singapore (Hybrid). Association for Computational Linguistics. ↗
  9. Areeb Alowisheq, and Haidar Khan. 2024. AL-LaM: Large language models for Arabic and English. Preprint, arXiv:2407.15390. ↗
  10. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  11. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279–6292, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  12. Google. 2024. Gemini 3 Flash model card. Google. find
  13. Google Cloud. 2025. Gemini 2.5 pro — generative ai on vertex ai. https://cloud.google.com/ vertex-ai/generative-ai/docs/models/gemini/ 2-5-pro. Last accessed: 2025-10-06. ↗
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. ArXiv preprint, abs/2407.21783. ↗
  15. Ahmed Heakl, Sara Ghaboura, Omkar Thawakar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, and Salman H. Khan. 2025a. AIN: The Arabic inclusive large multimodal model. ArXiv preprint, abs/2502.00094. ↗
  16. Ahmed Heakl, Muhammad Abdullah Sohail, Mukul Ranjan, Rania Elbadry, Ghazi Shazan Ahmad, Mohamed El-Geish, Omar Maher, Zhiqiang Shen, Fahad Shahbaz Khan, and Salman Khan. 2025b. KITAB-Bench: A comprehensive multi-domain benchmark for Arabic OCR and document understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22006–22024, Vienna, Austria. Association for Computational Linguistics. ↗
  17. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, et al. 2024. GPT-4o system card. Preprint, arXiv:2410.21276. ↗
  18. Islamic Financial Services Board (IFSB). 2024. Islamic financial services industry stability report 2024. find
  19. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. ArBanking77: Intent detection neural model and a new Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. Preprint, arXiv:2401.04088. ↗
  20. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, et al. 2025. Gemma 3 technical report. ArXiv preprint, abs/2503.19786. ↗
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018a. WWW’18 Open Challenge: Financial opinion mining and question answering. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW’18, pages 1941–1942, Lyon , France. ACM. ↗
  23. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018b. WWW’18 Open Challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. ↗
  24. Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, Yunhe Wang, and Dacheng Tao. 2025. CFinBench: A comprehensive Chinese financial benchmark for large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL’25, pages 876–891, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  25. OpenAI. 2025. GPT-5 system card. OpenAI. find
  26. Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, and Sophia Ananiadou. 2025a. Plutus: Benchmarking large language models in low-resource Greek finance. ArXiv preprint, abs/2502.18772. ↗
  27. Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Guojun Xiong, Zhiyang Deng, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E. Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, and Qianqian Xie. 2025b. MultiFinBen: A multilingual, multimodal, and difficulty-aware benchmark for financial LLM evaluation. ArXiv preprint, abs/2506.14028. ↗
  28. Felix Pomeranz. 1997. The accounting and auditing organization for Islamic financial institutions: An important regulatory debut. Journal of International Accounting, Auditing and Taxation, 6(1):123–130. ↗
  29. Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv preprint, abs/2408.00118. ↗
  30. Saudi Central Bank. 2024. Saudi Central Bank (sama) regulatory framework for Islamic finance. find
  31. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric P. Xing. 2023. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ArXiv preprint, abs/2308.16149. silma-ai. 2024. SILMA 9B Instruct v1.0. https://huggingface.co/silma-ai/ SILMA-9B-Instruct-v1.0. ↗
  32. Falcon-LLM Team. 2025. Falcon-Arabic: A breakthrough in Arabic language models. find
  33. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. FinBen: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems, 37:95716–95743. find
  34. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. PIXIU: A large language model, instruction data and evaluation benchmark for finance. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NeurIPS’23, Red Hook, NY, USA. Curran Associates Inc. ↗
  35. Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Jinyan Su, Aaryamon-vikram Singh, Rui Xing, Rania Elbadry, Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty, Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou, and Preslav Nakov. 2025. FinChain: A symbolic benchmark for verifiable chain-of-thought financial reasoning. Preprint, arXiv:2506.02515. ↗
  36. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. ArXiv preprint, abs/2412.15115. ↗
  37. Xiao Zhang, Ruoyu Xiang, Chenhan Yuan, Duanyu Feng, Weiguang Han, Alejandro Lopez-Lira, Xiao-Yang Liu, Meikang Qiu, Sophia Ananiadou, Min Peng, Jimin Huang, and Qianqian Xie. 2024. Dólares or Dollars? Unraveling the bilingual prowess of financial LLMs between Spanish and English. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, August 25-29, 2024, KDD’24, pages 6236– 6246, Barcelona, Spain. ACM. find
  38. Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024. Finance-MATH: Knowledge-intensive math reasoning in finance domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL’24, pages 12841–12858, Bangkok, Thailand. Association for Computational Linguistics. find
  39. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277–3287, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elbadry-2026-sahm-benchmark-arabic-financial-shari/