Elamur Research⌕
☰
home / papers / 2025 / Evaluation of Large Language Models on Arabic Punctuation Prediction

Evaluation of Large Language Models on Arabic Punctuation Prediction

Paper · 2025 · Asma Ali Al Wazrah, Afrah Altamimi, Hawra Aljasim et al.

Summary

The paper asks how well current large language models restore punctuation in Arabic text. This task matters for cleaning up speech-recognition output and for downstream translation, and had not been tested with LLMs for Arabic. The KSAA team assembled a fresh dataset for predicting where punctuation belongs in Arabic text, containing 10,046 data points sourced from books and spanning six punctuation marks. They tested seven LLMs (GPT4-o, Gemini1.5, JAIS, AceGPT, SILMA, ALLaM, CommandR+) with zero-shot and few-shot prompting. As a baseline they fine-tuned AraBERT on the training portion. AraBERT is strong on some individual marks. The general-purpose LLMs already perform respectably without examples and improve when given a few. GPT4-o and Gemini1.5 generalise best. A qualitative error analysis explains where scores mislead: models sometimes repair the source text's own mistakes, borrow punctuation habits from other languages, miss rare marks, and stumble on partially diacritised words. The work gives Arabic a benchmark for this task and a list of concrete failure types to target.

Key points

  • The corpus has 10,046 data points, with 962 for validation and 515 for testing. Each book is split 90/10 into train and validation, and one whole book is reserved only for testing.
  • The evaluated models are AraBERT v0.2 (fine-tuned), GPT4-o, Gemini1.5, JAIS-13b, AceGPT-13b, SILMA v1.0 (9B, based on Gemma), ALLaM-1 (13B, SDAIA), and Command R+ (104B, Cohere). The LLMs were run zero-shot and few-shot over six punctuation marks.
  • GPT4-o and Gemini1.5 were the most robust in both prompting settings. Few-shot examples generally helped the LLMs.
  • Rare marks were hard for every model. The exclamation mark appeared only five times in the test data, three of them combined with a question mark (؟!), and the semicolon was also rare.
  • Partially diacritised words disrupted AraBERT. It was more accurate when fine-tuned on data that kept the diacritics than when they were removed.

Interesting ideas

  • I1. Reserving an entire book for testing, instead of sampling sentences, keeps an author's style and topic from leaking into evaluation, so the scores reflect transfer to unseen writing.
  • I2. The authors note that some "errors" are actually the model correcting a flawed reference, so a perfect score may only mean the model copied the source's mistakes.
  • I3. Multilingual models carried punctuation conventions from other languages into Arabic, such as semicolon-separated lists replacing the colon-introduced structure the source used. This kind of cross-linguistic interference is worth checking for explicitly.
  • I4. Inconsistent surface marking (partial diacritics) caused failures that look like model weakness but are really input-normalisation problems.
  • I5. Reporting results per mark, with error categories, exposed rare and compound cases (such as ؟!) that a single overall score would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Hold out whole documents or genres (e.g. a full fable collection) when evaluating parsers and readers, so that memorised house style cannot inflate the results.
  • I2 (learn): When several witness parsers agree against the gold annotation, route the case to a "suspected gold error" queue instead of counting it as a failure. This also yields candidates for treebank repair.
  • I3 (learn): Audit outputs from the LLM teacher tiers for conventions borrowed from other UD dialects or registers before accepting them as silver data.
  • I4 (tokens): Detect and normalise mixed markup (quote styles, hyphen and dash variants, stray accents) before tokenizing and tagging. Log each normalisation so the step stays explainable.
  • I5 (answer): Break test results down by question type and by rare constructions, so that weak spots show up instead of being averaged away.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Asma Ali Al Wazrah, Afrah Altamimi, Hawra Aljasim, Waad Alshammari, Rawan Al-Matham, Omar Elnashar, Mohamed Amin and Abdulrahman AlOsaimy. 2025. Evaluation of Large Language Models on Arabic Punctuation Prediction. Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wazrah2025evaluation,
  title = {{Evaluation of Large Language Models on Arabic Punctuation Prediction}},
  author = {Asma Ali {Al Wazrah}, Afrah Altamimi, Hawra Aljasim, Waad Alshammari, Rawan Al-Matham, Omar Elnashar, Mohamed Amin and Abdulrahman AlOsaimy},
  booktitle = {Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script},
  year = {2025},
  url = {https://aclanthology.org/2025.abjadnlp-1.15/}
}

References

41 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelkarim Aboutaib, Ahmad El allaoui, Imad Zeroual, and El Wardani Dadi. 2023. Punctuation Prediction for the Arabic language. In Proceedings of the 6th International Conference on Networking, Intelligent Systems & Security, New York, NY, USA. Association for Computing Machinery. find
  2. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection. In Nizar Habash, Houda Bouamor, Hazem Hajj, Walid Magdy, Wajdi Zaghouani, Fethi Bougares, Nadi Tomeh, Ibrahim Abu Farha, and Samia Touileb, editors, Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  3. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual Evaluation of Generative AI. ↗
  4. Ali Saleh Alammary. 2022. BERT Models for Arabic Text Classification: A Systematic Review. Applied Sciences, 12(11). ↗
  5. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa Evaluation Benchmark for Arabic Language Models. In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham, editors, Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.
  6. Zaid Alyafeai, Maged S. Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models. ArXiv, abs/2306.16322. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  8. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. ↗
  9. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, et al. 2024. ALLaM: Large Language Models for Arabic and English. ↗
  10. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of 152 9 Artificial General Intelligence: Early experiments with GPT-4. Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  11. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023a. Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation. In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham, editors, Proceedings of ArabicNLP 2023, pages 232–243, Singapore (Hybrid). Association for Computational Linguistics.
  12. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine.
  13. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023b. ORCA: A Challenging Benchmark for Arabic Language Understanding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
  14. Jessica Nayeli López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane. 2023. GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts. In Karima Kadaoui, Samar M. Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties.
  15. Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist.
  16. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP. ArXiv, abs/2305.14976. ↗
  17. Sang Yun Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction. ArXiv, abs/2312.08400. ↗
  18. Aidan Gomez. 2024. Introducing Command R+: A Scalable LLM Built for Business. find
  19. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning.
  20. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023a. Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting.
  21. Md Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets.
  22. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023b. AceGPT, Localizing Large Language Models in Arabic. ArXiv, abs/2309.12053. ↗
  23. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023. Chain-of-Dictionary Prompting Elicits Translation in Large Language Models.
  24. Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to Design Translation Prompts for ChatGPT: An Empirical Study. find
  25. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing Youssef Mansour, Ashraf Elnagar, and Sane Yagi. 2023. Punctuation Prediction for the Arabic Language. In Abhishek Swaroop, Vineet Kansal, Giancarlo Fortino, and Aboul Ella Hassanien, editors, Proceedings of Fourth Doctoral Symposium on Computational Intelligence, volume 726 of Lecture Notes in Networks and Systems, pages 579–592. Springer Nature Singapore, Singapore. 153 10
  26. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed El-Shangiti, and Muhammad Abdul-Mageed. 2023. Dolphin: A Challenging and Diverse Benchmark for Arabic NLG.
  27. Oluwatosin Ogundare and Gustavo Quiros Araya. 2023. Comparative Analysis of CHATGPT and the evolution of language models. ↗
  28. OpenAI. 2024. Hello GPT-4o. find
  29. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards Making the Most of ChatGPT for Machine Translation. ↗
  30. Sundar Pichai and Demis Hassabis. 2024. Our next-generation model: Gemini 1.5. find
  31. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
  32. Xueyang Wu, Su Zhu, Yue Wu, and Kai Yu. 2016. Rich punctuations prediction using large-scale deep learning. 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP):1–5. find
  33. Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Ye Bai, and Cunhang Fan. 2020. Focal Loss for Punctuation Prediction. In Interspeech. find
  34. Zhikai Zhou, Tian Tan, and Yanmin Qian. 2022. Punctuation Prediction for Streaming On-Device Speech Recognition. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP):7277–7281. find
  35. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can Large Language Models Transform Computational Social Science? ↗
  36. Abdelrahman Sakr and Marwan Torki. 2023. AraPunc: Arabic Punctuation Restoration Using Transformers. 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA):1–6. find
  37. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, et al. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. ↗
  38. SILMA. 2024. Empowering Arabic Speakers with Cutting-Edge Generative AI Technologies. find
  39. Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks. In Conference on Empirical Methods in Natural Language Processing.
  40. Monica Sunkara, S. Ronanki, Kalpit Dixit, S. Bodapati, and Katrin Kirchhoff. 2020. Robust Prediction of Punctuation and Truecasing for Medical ASR. ArXiv, abs/2007.02025. ↗
  41. Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-Level Machine Translation with Large Language Models. 154 11 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wazrah-2025-evaluation-large-language-models-arabic/