Elamur Research⌕
☰
home / papers / 2025 / How well can LLMs Grade Essays in Arabic?

How well can LLMs Grade Essays in Arabic?

Summary

The paper asks whether current generative LLMs can mark university essays written in Arabic, a language where automated essay scoring (AES) has mostly relied on hand-crafted features. Using the AR-AES dataset of real student answers, the authors test ChatGPT-4, Llama 2, OpenLLaMA, Aya, Jais and ACEGPT under zero-shot, few-shot in-context and fine-tuned settings. They also add the marking guidelines to prompts and try a bilingual prompt, with English instructions wrapped around Arabic essays. To deal with the inflated token counts of Arabic text, they build an optimized SentencePiece tokenizer that shortens sequences and cuts memory use. Among the LLMs, ACEGPT does best overall (QWK 0.67), but the much smaller AraBERT still wins (QWK 0.88). Fine-tuning beats prompting, and prompt design matters a great deal when no training is done. The work matters because it is the first systematic multi-LLM benchmark of Arabic AES on authentic data. It also shows that, with small training sets, compact pretrained encoders remain the stronger baseline.

Key points

  • AR-AES contains 2,046 essays by undergraduates at Umm Al-Qura University. It covers 12 questions of three types: argumentative, narrative and source-dependent. Fine-tuning uses a 70/15/15 train/validation/test split.
  • On the full dataset, ACEGPT reached QWK 0.67, the best among the LLMs. AraBERT reached 0.88.
  • English instructions combined with Arabic content improved performance by 49.49% over prompts written entirely in Arabic or entirely in English.
  • Fine-tuning worked better than zero-shot or few-shot prompting, even with as few as 74 training examples. Two approaches were used:
    • label-supervised adaptation of Llama 2 7B (LS-LLaMA);
    • instruction fine-tuning of OpenLLaMA 3B.
  • Agreement with the course director exceeded human inter-rater agreement on 8 of 12 questions. For example, few-shot ChatGPT-4 scored QWK 0.93 on Question 8, where the human raters reached 0.78. Humans agreed more on source-dependent questions.

Interesting ideas

  • I1. Write the task instructions in the language the model handles best, and leave the material being judged in its original language; this bilingual split gave a large gain cheaply. — For Mova (learn): Prompts to the LLM teacher tiers could keep one fixed, well-tested English instruction template while passing raw text untouched, so that changes in teacher labels reflect the data rather than the wording of the prompt.
  • I2. Put the explicit marking rubric inside the scoring instruction, so that grading follows stated criteria instead of a vague overall impression. — For Mova (answer): School-course test grading could check a student answer against a list of rubric items, each matched to expected frames or entities, which makes every awarded point explainable.
  • I3. Use human inter-rater agreement for each question as a reference ceiling, and break results down by question type, objective versus subjective. — For Mova (learn): Evaluation could report where Mova's agreement sits relative to annotator agreement on the same items, separating genuine errors from cases that are inherently ambiguous.
  • I4. Report a tolerant "within one grade" accuracy alongside QWK, so that near misses are distinguished from wild errors. — For Mova (answer): For maths word problems and QA scoring, a closeness measure, such as an off-by-one quantity or a partially correct role, would show whether failures are small slips or real misunderstandings of the text world.
  • I5. Retrain the subword tokenizer on in-domain text to stop over-segmentation, which shortens sequences and lowers compute cost. — For Mova (tree): The gparse transformer parser could use a vocabulary fitted to its treebanks and target domains, keeping inputs short and word pieces aligned with UD tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.16516 · arXiv 2501.16516

@misc{ghazawi2025how,
  title = {{How well can LLMs Grade Essays in Arabic?}},
  author = {Ghazawi, Rayed and Simpson, Edwin},
  year = {2025},
  doi = {10.48550/arxiv.2501.16516},
  eprint = {2501.16516},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.16516}
}

References

58 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yang, K., Raković, M., Li, Y., Guan, Q., Gašević, D., Chen, G.: Unveiling the tapestry of automated essay scoring: A comprehensive investigation of accuracy, fairness, and generalizability. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 22466–22474 (2024) find
  2. Silveira, I.C., Barbosa, A., Mauá, D.D.: A new benchmark for automatic essay scoring in Portuguese. In: Proceedings of the 16th International Conference on Computational Processing of Portuguese (2024). https://github.com/kamel-usp/aes_enem ↗
  3. Franci, Y., Franci, Y.A.: Enhancing Automated Essay Evaluation: The Impact of Advanced Generative Pre-trained Transformers on Educational Feedback (2023). https://doi.org/10.13140/RG.2.2.13615.51365/1 . https://www. researchgate.net/publication/376271077 ↗
  4. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., Ge, B.: Summary of ChatGPT-related research and perspective towards the future of Large Language Models. Meta-Radiology 1, 100017 (2023) https://doi.org/10. 1016/j.metrad.2023.100017 ↗ our card
  5. Ahmed, T., Devanbu, P.: Few-shot training LLMs for project-specific code-summarization. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–5 (2022) find
  6. Xiao, C., Ma, W., Xu, S.X., Zhang, K., Wang, Y., Fu, Q.: From automation to augmentation: Large Language Models elevating essay scoring landscape. arXiv preprint arXiv:2401.06431 (2024) ↗
  7. Mansour, W.A., Albatarni, S., Eltanbouly, S., Elsayed, T.: Can Large Language Models automatically score proficiency of written essays? In: Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 2777–2786. ELRA and ICCL, Torino, Italia (2024). https://aclanthology.org/2024.lrec-main.247 ↗
  8. Ghazawi, R., Simpson, E.: Automated essay scoring in Arabic: a dataset and analysis of a Bert-based system. arXiv preprint arXiv:2407.11212 (2024) ↗ our card
  9. Page, E.B.: The imminence of... grading essays by computer. The Phi Delta Kappan 47(5), 238–243 (1966) find
  10. Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 180–189 (2011) find our card
  11. Chen, H., He, B.: Automated essay scoring by maximizing human-machine agreement. Association for Computational Linguistics, 1741–1752 (2013) find
  12. Phandi, P., Chai, K.M.A., Ng, H.T.: Flexible domain adaptation for automated essay scoring using correlated linear regression. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 431–439 (2015) find
  13. Dong, F., Zhang, Y., Yang, J.: Attention-based recurrent convolutional neural network for automatic essay scoring. In: Levy, R., Specia, L. (eds.) Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 153–162. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/K17-1017 . https://aclanthology.org/K17-1017 ↗ our card
  14. Li, X., Yang, H., Hu, S., Geng, J., Lin, K., Li, Y.: Enhanced hybrid neural network for automated essay scoring. Expert Systems 39(10), 13068 (2022) find
  15. Chen, M., Li, X.: Relevance-based automated essay scoring via hierarchical recurrent model. In: 2018 International Conference on Asian Language Processing (IALP), pp. 378–383 (2018). IEEE find
  16. Wang, Y., Wang, C., Li, R., Lin, H.: On the use of BERT for automated essay scoring: Joint learning of multi-scale essay representation. arXiv preprint arXiv:2205.03835 (2022) ↗
  17. Yang, R., Cao, J., Wen, Z., Wu, Y., He, X.: Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1560–1569 (2020) find
  18. Machhout, R.A., Zribi, C.B.O., Bouzid, S.M.: Arabic automatic essay scoring systems: An overview study. In: International Conference on Intelligent Systems Design and Applications, pp. 1164–1176 (2021). Springer find
  19. Alghamdi, M., Alkanhal, M., Al-Badrashiny, M., Al-Qabbany, A., Areshey, A., Alharbi, A.: A hybrid automatic scoring system for Arabic essays. AI Communications 27(2), 103–111 (2014) https://doi.org/10.3233/AIC-130586 ↗
  20. Al-Shalabi, E.F.: An automated system for essay scoring of online exams in Arabic based on stemming techniques and levenshtein edit operations. International Journal of Computer Science Issues 13(5), 45–50 (2016) find
  21. Alqahtani, A., Alsaif, A.: Automated Arabic essay evaluation. In: Proceedings of the 17th International Conference on Natural Language Processing (ICON), pp. 181–190. NLP Association of India (NLPAI), Indian Institute of Technology Patna, Patna, India (2020). https://aclanthology.org/2020.icon-main.24 ↗
  22. Alqahtani, A., Alsaif, A.: Automatic Evaluation for Arabic Essays: A Rule-Based System. 2019 IEEE international symposium on signal processing and information technology (ISSPIT) (2019) find
  23. Al-Shargabi, B., Alzyadat, R., Hamad, F.: AEGD: Arabic essay grading dataset for machine learning. Journal of Theoretical and Applied Information Technology (2021) find
  24. Gaheen, M.M., ElEraky, R.M., Ewees, A.A.: Automated students Arabic essay scoring using trained neural network by E-Jaya optimization to support personalized system of instruction. Education and Information Technologies 26(1), 1165–1181 (2021) https://doi.org/10.1007/s10639-020-10300-6 ↗
  25. Ramesh, D., Sanampudi, S.K.: An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review 55(3), 2495–2527 (2022) find
  26. Krugmann, J.O., Hartmann, J.: Sentiment analysis in the age of generative AI. Customer Needs and Solutions 11(1), 3 (2024) find
  27. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) ↗
  28. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) ↗
  29. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O.M., Kamboj, S., Pandit, O., Pal, R., et al.: Jais and Jais-Chat: Arabic-centric foundation and instruction-tuned open generative Large Language Models. arXiv preprint arXiv:2308.16149 (2023) ↗ our card
  30. Ammar, A., Koubaa, A., Benjdira, B., Nacar, O., Sibaee, S.: Prediction of Arabic legal rulings using large language models. Electronics 13(4), 764 (2024) find our card
  31. Tamer, A., Hassan, A.-A., Ali, A., Salah, N., Medhat, W.: Fine tuning of Large Language Models for Arabic Language. In: 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pp. 1–4 (2023). IEEE find
  32. Katuka, G.A., Gain, A., Yu, Y.-Y.: Investigating automatic scoring and feedback using Large Language Models. arXiv preprint arXiv:2405.00602 (2024) ↗
  33. Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., Kasneci, G.: ChatGPT for good? on opportunities and challenges of Large Language Models for education. Learning and Individual Differences 103, 102274 (2023) https://doi.org/10.1016/j.lindif. 2023.102274 ↗ our card
  34. Latif, E., Zhai, X.: Fine-tuning ChatGPT for automatic scoring. Computers and Education: Artificial Intelligence 6, 100210 (2024) find
  35. Henkel, O., Hills, L., Roberts, B., McGrane, J.: Can llms grade open response reading comprehension questions? an empirical study using the roars dataset. International Journal of Artificial Intelligence in Education, 1–26 (2024) find
  36. Henkel, O., Hills, L., Boxer, A., Roberts, B., Levonian, Z.: Can large language models make the grade? an empirical study evaluating LLMs’ ability to mark short answer questions in k-12 education. In: Proceedings of the Eleventh ACM Conference on Learning@ Scale, pp. 300–304 (2024) find
  37. Rasul, T., Nair, S., Kalendra, D., Robin, M., Oliveira Santini, F., Ladeira, W.J., Sun, M., Day, I., Rather, R.A., Heathcote, L.: The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning and Teaching 6(1) (2023) find
  38. Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A.B., Alraeesi, A., Almubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., et al.: ArabicMMLU: Assessing massive multitask language understanding in Arabic. arXiv preprint arXiv:2402.12840 (2024) ↗ our card
  39. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023) ↗ our card
  40. Ghosn, Y., El Sardouk, O., Jabbour, Y., Jrad, M., Hussein Kamareddine, M., Abbas, N., Saade, C., Abi Ghanem, A.: ChatGPT 4 versus ChatGPT 3.5 on the final FRCR part a sample questions. assessing performance and accuracy of explanations. medRxiv, 2023–09 (2023) find
  41. Massey, P.A., Montgomery, C., Zhang, A.S.: Comparison of ChatGPT–3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations. JAAOS-Journal of the American Academy of Orthopaedic Surgeons 31(23), 1173–1179 (2023) find
  42. Plevris, V., Papazafeiropoulos, G., Jiménez Rios, A.: Chatbots put to the test in math and logic problems: a comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. AI 4(4), 949–969 (2023) find
  43. Üstün, A., Aryabumi, V., Yong, Z.-X., Ko, W.-Y., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., et al.: Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827 (2024) ↗ our card
  44. Alyafeai, Z., Masoud, M., Ghaleb, M., Al-shaibani, M.S.: Masader: Metadata sourcing for Arabic text and speech data resources. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., Piperidis, S. (eds.) Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6340–6351. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.lrec-1.681 ↗ our card
  45. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M.S., Shen, S., Yong, Z.X., Schoelkopf, H., Tang, X., Radev, D., Aji, A.F., Almubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff, E., Raffel, C.: Crosslingual generalization through multitask finetuning. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15991–16111. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10.18653/v1/2023.acl-long.891 . https://aclanthology.org/2023.acl-long.891 ↗ our card
  46. Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Dingjie, S., Chen, Z., Alharthi, M., An, B., He, J., Liu, Z., Chen, J., Li, J., Wang, B., Zhang, L., Sun, R., Wan, X., Li, H., Xu, J.: AceGPT, localizing large language models in Arabic. Human Language Technologies (Volume 1: Long Papers), 8139–8163 (2024) https://doi.org/10.18653/v1/2024.naacl-long.450 ↗ our card
  47. Demidova, A., Atwany, H., Rabih, N., Sha’ban, S.: Arabic train at NADI 2024 shared task: LLMs’ ability to translate Arabic dialects into Modern Standard Arabic. In: Habash, N., Bouamor, H., Eskander, R., Tomeh, N., Abu Farha, I., Abdelali, A., Touileb, S., Hamed, I., Onaizan, Y., Alhafni, B., Antoun, W., Khalifa, S., Haddad, H., Zitouni, I., AlKhamissi, B., Almatham, R., Mrini, K. (eds.) Proceedings of The Second Arabic Natural Language Processing Conference, pp. 729–734. Association for Computational Linguistics, Bangkok, Thailand (2024). https://doi.org/10.18653/v1/2024.arabicnlp-1.80 . https://aclanthology.org/2024.arabicnlp-1.80 ↗
  48. Mathias, S., Bhattacharyya, P.: ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018) find
  49. Bari, M.S., Alnumay, Y., Alzahrani, N.A., Alotaibi, N.M., Alyahya, H.A., Al-Rashed, S., Mirza, F.A., Alsubaie, S.Z., Alahmed, H.A., Alabduljabbar, G., et al.: ALLaM: Large Language Models for Arabic and English. arXiv preprint arXiv:2407.15390 (2024) ↗ our card
  50. Petrov, A., La Malfa, E., Torr, P., Bibi, A.: Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems 36 (2024) find
  51. Kwon, S., Bhatia, G., Abdul-Mageed, M., et al.: Beyond English: Evaluating LLMs for Arabic grammatical error correction. In: Proceedings of ArabicNLP 2023, pp. 101–119 (2023) find
  52. Patil, R., Gudivada, V.: A review of current trends, techniques, and challenges in Large Language Models (LLMs). Applied Sciences 14(5), 2074 (2024) find
  53. Edwards, A., Camacho-Collados, J.: Language models for text classification: Is in-context learning enough? arXiv preprint arXiv:2403.17661 (2024) ↗
  54. Chamieh, I., Zesch, T., Giebermann, K.: LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches. In: Kochmar, E., Bexte, M., Burstein, J., Horbach, A., Laarmann-Quante, R., Tack, A., Yaneva, V., Yuan, Z. (eds.) Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pp. 309–315. Association for Computational Linguistics, Mexico City, Mexico (2024). https://aclanthology.org/2024.bea-1.25 ↗
  55. Gema, A., Daines, L., Minervini, P., Alex, B.: Parameter-efficient fine-tuning of Llama for the clinical domain. arXiv preprint arXiv:2307.03042 (2023) ↗
  56. Li, Z., Li, X., Liu, Y., Xie, H., Li, J., Wang, F.-l., Li, Q., Zhong, X.: Label supervised Llama finetuning. arXiv preprint arXiv:2310.01208 (2023) ↗
  57. Geng, X., Liu, H.: OpenLLaMA: An Open Reproduction of LLaMA (2023). https://github.com/openlm-research/open_llama ↗
  58. Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al.: ChatGPT: Jack of all trades, master of none. Information Fusion 99, 101861 (2023) find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghazawi-2025-well-llms-grade-essays-arabic/