Elamur Research⌕
☰
home / papers / 2026 / The Arabic Bible as an Evaluation Tool: The Case Study of the Khalīlī Arabic Dialect

The Arabic Bible as an Evaluation Tool: The Case Study of the Khalīlī Arabic Dialect

Paper · 2026 · Jakub Zbrzeżny, Ehud Reiter and Wei Zhao

Summary

Colloquial Arabic is poorly covered in NLP, especially at the level of individual cities or rural districts. Bible translations are a rarely used and little-documented source for such varieties. This exploratory study treats a Bible rendition as an evaluation resource for the rural Khalīlī dialect of the Hebron area.

The data come from an unpublished paraphrase of Genesis and Matthew into rural Khalili, produced by native speakers at the University of Aberdeen. It was kept out of model training through API access. Three LLMs were asked to translate English or MSA passages into the dialect and to render dialect passages into MSA. Three native-speaker evaluators then scored the anonymised outputs on a dialect-specificity scale.

No model produced genuinely Khalili text. The best outputs reached only a general Palestinian level, and most read as broadly Levantine. In contrast, translation from the dialect into MSA was judged highly accurate. The models therefore comprehend local dialects much better than they generate them. The authors credit expert, multidisciplinary collaboration for the reliability of these results.

Key points

  • Data: a dialect corpus of about 40,000 words (Genesis and Matthew) from the Hexapla Arabica project.
    • English references were the JPS 1917 text for Genesis and the OpenEnglishBible for Matthew.
    • MSA came from the Van Dyck (1865) translation in its STEP-Bible edition.
  • Models and outputs: Gemini 2 Flash, ChatGPT 4o mini and ChatGPT 4o produced 200 dialect units (about 50,000 words) and 80 MSA units (about 22,000 words).
    • Translation directions were E>D, E>MSA>D and MSA>D, plus example-based prompts for GPT4o.
  • Scoring scale (0–100):
    • 100 = rural Khalili
    • 75 = Khalili
    • 50 = Palestinian
    • 25 = Levantine
    • 0 = generic Colloquial Arabic
  • Dialect-generation results: Gem2F scored highest with 57 (MSA>D) and 54 (E>MSA>D), which corresponds to the Palestinian level. The lowest score was 15 (GPT4om, MSA>D).
  • Dialect-to-MSA results: accuracy was 95–97 for all models.
    • The outputs preserved distinctive features of the paraphrase, such as its unusual wording of the Lord's Prayer.
    • The authors caution that LLM-driven tools may accelerate dialect levelling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 worksReligious studies · 227 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Jakub Zbrzeżny, Ehud Reiter and Wei Zhao. 2026. The Arabic Bible as an Evaluation Tool: The Case Study of the Khalīlī Arabic Dialect. Proceedings of the 1st Symposium on Natural Language Generation Evaluations.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zbrzezny2026arabic,
  title = {{The Arabic Bible as an Evaluation Tool: The Case Study of the Khalīlī Arabic Dialect}},
  author = {Jakub Zbrzeżny, Ehud Reiter and Wei Zhao},
  booktitle = {Proceedings of the 1st Symposium on Natural Language Generation Evaluations},
  year = {2026},
  url = {https://aclanthology.org/2026.retroeval-main.4/}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karim Al-Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 769–778, Marseille, France. European Language Resources Association. Ethical Statement All due considerations (i.a. religious, social, cultural, political, ethical, economic) were undertaken prior to the commencement of the research to ensure that the safety of the evaluators was not put at risk, and that their work was appropriately remunerated. ↗
  2. Abdullah Alabdullah, Lifeng Han, and Chenghua Lin. 2025. Advancing dialectal arabic to modern standard arabic machine translation. Preprint, arXiv:2507.20301. ↗
  3. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association. find
  4. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader, and Anis Koubaa. 2025. Towards inclusive arabic LLMs: A culturally aligned benchmark in arabic large language model evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 387–401, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  5. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
  6. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal arabic-english machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  7. Alwajih Fakhraddin, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. Palm: A culturally inclusive and linguistically diverse dataset for arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics.
  8. Kees Versteegh, editor. 2005–2007. Encyclopedia of Arabic Language and Linguistics. Brill. find
  9. Stefan Weninger, Geoffrey Khan, Michael P. Streck, and Janet C. E. Watson, editors. 2012. Semitic Languages: An International Handbook. De Gruyter. find
  10. Jezia Zakraoui, Moutaz Saleh, Somaya Al-Maadeed, and Jihad M. Alja’am. 2021. Arabic machine translation: A survey with challenges and future directions. IEEE Access, 9:161445–161468. ↗
  11. Salima Harrat, Karima Meftouh, and Kamel Smaili. 2019. Machine translation for arabic dialects (survey). Information Processing Management, 56(2):262–273. Advance Arabic Natural Language Processing (ANLP) and its Applications. find
  12. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine translation of arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT ’12), pages 49–59, USA. Association for Computational Linguistics.
  13. Karima Kadaoui, Samar M. Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. TARJAMAT: Evaluation of bard and ChatGPT on machine translation of ten arabic varieties. In Proceedings of ArabicNLP 2023, pages 52–75, Singapore (Hybrid). Association for Computational Linguistics.
  14. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 220–247, Singapore. Association for Computational Linguistics.
  15. Abdellah El Mekki, Samar M. Magdy, Houdaifa Atou, Ruwa AbuHweidi, Baraah Qawasmeh, Omer Nacar, Thikra Al-hibiri, Razan Saadie, Hamzah Alsayadi, Nadia Ghezaiel Hammouda, Alshima Alkhazimi, Aya Hamod, Al-Yas Al-Ghafri, Wesam El-Sayed, Asila Al sharji, Mohamad Ballout, Anas Belfathi, Karim Ghaddar, Serry Sibaee, and 28 others. 2026. Alexandria: A multi-domain dialectal arabic machine translation dataset for culturally inclusive and linguistically diverse llms. Preprint, arXiv:2601.13099. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zbrzezny-2026-arabic-bible-as-evaluation-tool/