Elamur Research⌕
☰
home / papers / 2026 / LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring

LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring

Paper · 2026 · May Bashendy, Walid Massoud, Sohaila Eltanbouly et al.

Summary

Arabic automated essay scoring (AES) has lagged behind English because few public Arabic datasets exist. Those that do are small, have no per-trait labels, or have no labels at all. The authors address this gap with LAILA. Over one academic year, they gathered 7,859 essays from 4,372 students in 24 Qatari high schools (grades 10–12). Students wrote on 8 prompts under timed, supervised conditions using a digital platform. The work follows nine stated design principles (D1–D9) covering:

  • authenticity of the writing,
  • diversity of schools and prompts,
  • gender balance,
  • full trait coverage,
  • privacy,
  • standardized annotation.

Besides an overall holistic mark, every essay is rated on seven separate traits: grammar, mechanics, development, style, vocabulary, organization, and relevance. The authors report substantial inter-annotator agreement. They benchmark Arabic and English models, including LLMs such as Fanar and R7B used with few-shot prompting, in both prompt-specific and cross-prompt settings. The best systems still fall well short of human raters, especially on unseen prompts. The dataset, guidelines and baseline code are public, which gives Arabic AES its first benchmark at a scale comparable to English resources.

Key points

  • LAILA has 7,859 essays on 8 prompts from 4,372 students in grades 10–12. The average essay length is about 171 words. It is described as the largest public Arabic AES dataset.
  • Labels cover a holistic score plus seven traits. Relevance uses a narrow 0–2 scale, and this coarse scale likely depresses its QWK.
  • LAILA fully includes the TAQEEM 2025 shared-task data (4 prompts, 1,265 essays). It reuses two prompt texts from QAES but none of its essays.
  • Annotation was done by 6 annotators and 3 supervisors working from a guidebook and trait rubrics. Disagreements were settled by adjudication, and annotators could not see student identities.
  • The data and baselines are released on GitLab (bigirqu/laila, laila-baselines) for research use only. Known limits:
    • one country;
    • only explanatory and persuasive genres;
    • two prompts with fewer essays than the others.

Interesting ideas

  • I1. Breaking one overall quality judgment into separately scored traits (organization, grammar, development, etc.) yields diagnosable feedback rather than a single opaque number.
  • I2. Reporting both a within-prompt setup and a cross-prompt setup (train on some tasks, test on entirely unseen ones) exposes how much a model relies on task-specific patterns.
  • I3. The authors note that a very coarse label scale (0–2) can make an agreement metric like QWK understate model quality. Metric choice must therefore match label granularity.
  • I4. Collecting writing digitally under timed, unassisted conditions keeps genuine learner errors and avoids transcription noise.
  • I5. Annotation used blinded raters plus supervisor adjudication, following a shared guidebook and rubric per trait.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course test answers could be graded along separate axes, such as factual correctness, use of the right entities and coherence, so a failure points to the responsible stage.
  • I2 (learn): Evaluation could hold out whole sources or genres, such as unseen fables or new word-problem templates, to check that repair rules and trained parsers generalize rather than memorize.
  • I3 (learn): When scoring silver data or teacher agreement on low-cardinality labels (e.g., speech-act classes), pair kappa-style metrics with per-class error counts so that weak-looking scores are not misread.
  • I4 (tokens): A small test set of authentic, uncorrected learner English could check that the tokenizer, tagger and lemmatizer handle real misspellings and nonstandard punctuation without collapsing.
  • I5 (learn): LLM teacher tiers could follow the same pattern, with independent labelers blind to source metadata and a higher tier adjudicating only where they disagree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksGrammatical error correction & learner language · 162 works

Cite

Cite as May Bashendy, Walid Massoud, Sohaila Eltanbouly, Salam Albatarni, Marwan Sayed, Abrar Abir, Houda Bouamor and Tamer Elsayed. 2026. LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.142

@inproceedings{bashendy2026laila,
  title = {{LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring}},
  author = {May Bashendy, Walid Massoud, Sohaila Eltanbouly, Salam Albatarni, Marwan Sayed, Abrar Abir, Houda Bouamor and Tamer Elsayed},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.142},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.142}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hikmat A Abdeljaber. 2021. Automatic Arabic short answers scoring using longest common subsequence and Arabic WordNet. IEEE Access, 9:76433–76445. ↗
  2. Abdelhamid M. Ahmed, Xiao Zhang, Lameya M. Rezk, and Wajdi Zaghouani. 2024. Building an Annotated L1 Arabic/L2 English Bilingual Writer Corpus: The Qatari Corpus of Argumentative Writing (QCAW). Corpus-based Studies across Humanities, 1(1):183– 215. ↗
  3. Mansour Alghamdi, Mohamed Alkanhal, Mohamed Al-Badrashiny, Abdulaziz Al-Qabbany, Ali Areshey, and Abdulaziz Alharbi. 2014. A hybrid automatic scoring system for Arabic essays. AI Communications, 27(2):103–111. ↗
  4. Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, and Jennifer Tracey. 2025. Command R7B Arabic: A small, enterprise focused, multilingual, and culturally aware Arabic LLM. ↗
  5. Mohammad Alobed, Abdallah M M Altrad, and Zainab Binti Abu Bakar. 2021a. A comparative analysis of euclidean, jaccard and cosine similarity measure and Arabic wordnet for automated Arabic essay scoring. find
  6. In 2021 Fifth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 70–74. of the Second Arabic Natural Language Processing Conference, pages 337–351, Bangkok, Thailand. Association for Computational Linguistics. find
  7. Mohammad Alobed, Abdallah MM Altrad, Zainab Binti Abu Bakar, and Norshuhani Zamin. 2021b. Automated Arabic essay scoring based on hybrid stemming with WordNet. Malaysian Journal of Computer Science, pages 55–67. ↗
  8. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc. find
  9. Abeer Alqahtani and Amal Alsaif. 2020. Automated Arabic essay evaluation. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 181–190, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
  10. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. TOEFL11: A corpus of non-native English. ETS Research Report Series, 2013(2):i–15. ↗
  11. Waleed Alsanie, Mohamed I Alkanhal, Mohammed Al-hamadi, and Abdulaziz O Alqabbany. 2022. Automatic scoring of Arabic essays over three linguistic levels. Progress in Artificial Intelligence, pages 1– 13. ↗
  12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  13. Aqil M. Azmi, Maram F. Al-Jouie, and Muhammad Hussain. 2019. AAEE – automated evaluation of students’ essays in Arabic language. Information Processing & Management, 56(5):1736–1752. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. ↗
  14. Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2025. ALLaM: Large language models for Arabic and English. In The Thirteenth International Conference on Learning Representations.
  15. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Walid Massoud, Houda Bouamor, and Tamer Elsayed. 2025. TAQEEM 2025: Overview of the first shared task for Arabic quality evaluation of essays in multi-dimensions. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 966–976, Suzhou, China. Association for Computational Linguistics. find
  16. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud, and Houda Bouamor. 2024. QAES: First publicly-available trait-specific annotations for automated scoring of Arabic essays. In Proceedings Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Stindlová, and Chiara Vettori. 2014. The MERLIN corpus: Learner language and the CEFR. In LREC, pages 1281–1288. Reykjavik, Iceland.
  17. Po-Kai Chen, Bo-Wei Tsai, Shao Kuan Wei, Chien-Yao Wang, Jia-Ching Wang, and Yi-Ting Huang. 2025. Mixture of ordered scoring experts for cross-prompt essay trait scoring. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18071– 18084, Vienna, Austria. Association for Computational Linguistics. find
  18. Scott Crossley, Yu Tian, Perpetual Baffour, Alex Franklin, Youngmeen Kim, Wesley Morris, Meg Benner, Aigner Picou, and Ulrich Boser. 2023a. The English language learner insight, proficiency and skills evaluation (ELLIPSE) corpus. International Journal of Learner Corpus Research, 9(2):248–269. ↗
  19. Scott Andrew Crossley, Perpetual Baffour, Yu Tian, Alex Franklin, Meg Benner, and Ulrich Boser. 2023b. A large-scale corpus for assessing written argumentation: PERSUADE 2.0. Available at SSRN 4795747. ↗
  20. Heejin Do, Yunsu Kim, and Gary Lee. 2024. Autoregressive score generation for multi-trait essay scoring. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1659–1666, St. Julian’s, Malta. Association for Computational Linguistics. find
  21. Heejin Do, Yunsu Kim, and Gary Geunbae Lee. 2023. Prompt- and trait relation-aware cross-prompt essay trait scoring. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1538–1551, Toronto, Canada. Association for Computational Linguistics. find
  22. Yomna Elsayed, Emad Nabil, Marwan Torki, Safiullah Faizullah, and Ayman Khalafallah. 2025. ZaQQ: A new Arabic dataset for automatic essay scoring via a novel human–ai collaborative framework. Data, 10(9):148. ↗
  23. Marwa M. Gaheen, Rania M. ElEraky, and Ahmed A. Ewees. 2020. Optimized neural network-based improved multiverse optimizer algorithm for automated Arabic essay scoring. International Journal of Scientific & Technology Research, 9:238–243. find
  24. Rayed Ghazawi and Edwin Simpson. 2024. Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system. arXiv preprint arXiv:2407.11212. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. ↗
  25. Rayed Ghazawi and Edwin Simpson. 2025. How well can llms grade essays in Arabic? arXiv preprint arXiv:2501.16516. ↗
  26. Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  27. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  28. Marwan Sayed, Sohaila Eltanbouly, May Bashendy, and Tamer Elsayed. 2025. Feature engineering is not dead: A step towards state of the art for Arabic automated essay scoring. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 231–245, Suzhou, China. Association for Computational Linguistics.
  29. Yaqiong He, Feng Jiang, Xiaomin Chu, and Peifeng Li. 2022. Automated Chinese essay scoring from multiple traits. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3007–3016, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  30. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  31. Shengjie Li and Vincent Ng. 2024. Conundrums in cross-prompt automated essay scoring: Making sense of the state of the art. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7661– 7681, Bangkok, Thailand. Association for Computational Linguistics. ↗
  32. Rim Aroua Machhout and Chiraz Ben Othmane Zribi. 2024. Enhanced BERT approach to score Arabic essay’s relevance to the prompt. Communications of the IBIMA, 2024. ↗
  33. Somaia Mahmoud, Emad Nabil, and Marwan Torki. 2024. Automatic scoring of Arabic essays: A parameter-efficient approach for grammatical assessment. IEEE Access. ↗
  34. Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  35. Abu Bakr Mohammad, Kareem Eissa, and Samhaa El-Beltagy. 2017. AraVec: A set of Arabic word embedding models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  36. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar. Association for Computational Linguistics.
  37. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation.
  38. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An Arabic-centric multimodal generative AI platform. ↗
  39. Rodrigo Wilkens, Alice Pintard, David Alfter, Vincent Folny, and Thomas François. 2023. TCFLE-8: a corpus of learner written productions for French as a foreign language and its application to automated essay scoring. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3447–3465, Singapore. Association for Computational Linguistics. find
  40. David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1):2–13. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bashendy-2026-laila-large-trait-based-dataset/