Elamur Research⌕
☰
home / papers / 2024 / QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays

QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays

Paper · 2024 · May Bashendy, Salam Albatarni, Sohaila Eltanbouly et al.

Summary

Arabic automated essay scoring (AES) has lagged behind English mainly because no openly shared Arabic essays carry quality scores, and almost none carry scores per writing trait. The authors built QAES by annotating the Arabic part of the Qatari Corpus of Argumentative Writing (QCAW): 195 argumentative essays by first-year native-speaker university students, answering two TOEFL-derived prompts. Each essay received scores from two Arabic language specialists across seven traits, covering grammar and mechanics, vocabulary and style, as well as organization, development and relevance. They used the CAST rubrics from Qatar University's testing center, and a holistic score is obtained by adding up the trait scores. The authors also describe the annotation difficulties they met. As benchmarks, they ran a feature-based linear regression and two English state-of-the-art neural scorers, one single-trait (STL) and one multi-trait (MTL). The simple regression reached the best QWK, while the neural models usually had lower RMSE. The authors explain this through the very small data size. QAES is the first public resource of its kind for Arabic and a starting point for trait-level feedback to learners.

Key points

  • QAES covers 195 Arabic essays from QCAW: 115 for Task 1 (communication via phones and email) and 80 for Task 2 (technology and learning). Essays average about 489 words, with a range of 239–806, and the corpus has 95,342 tokens in total.
  • Relevance is scored 0–2. The other six traits are scored 1–5, and each level has a written descriptor. Every trait gets 0 if the essay was memorized, copied from the web or not attempted. HOL is the sum of the seven traits.
  • Baselines are a linear regression (LR) over n-gram, semantic, syntactic, lexical and surface features, plus English neural STL and MTL models.
  • LR gave the highest QWK, and MTL sometimes scored slightly below zero (e.g. -0.052 on GRA in Task 1). STL had the lowest RMSE on most Task 1 traits, and all models scored higher on Task 2.
  • A leave-one-category-out ablation found the following:
    • Dropping n-grams hurt nearly every Task 1 trait except REL.
    • Dropping surface features actually improved VOC and GRA.
    • Semantic features mattered for DEV.

Interesting ideas

  • I1. Defining the overall score as an explicit sum of rubric-defined trait scores keeps the final grade fully decomposable, so each point can be traced to a named criterion.
  • I2. Each score level comes with a concrete prose descriptor, such as "no intro/conclusion" versus "clear intro, 2–3 body paragraphs", which anchors judges to observable properties instead of impressions.
  • I3. Reporting QWK and RMSE side by side exposed a disagreement: the best-QWK model had the worst RMSE. This shows a single metric can mislead on tiny datasets.
  • I4. A hard gate gives zero to copied or non-attempted responses before any trait scoring happens, which separates validity checks from quality grading.
  • I5. Ablating whole feature families one at a time, separately per trait, showed that the same family can help one target and hurt another.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course test answers could be graded per criterion (relevance, correctness, completeness) and then summed, so every lost point has a stated reason.
  • I2 (learn): LLM teacher tiers could be given level-by-level descriptors when judging silver frames or answers, making their verdicts more consistent and auditable.
  • I3 (learn): Evaluation on small hand-checked sets should use at least two complementary metrics and flag cases where they rank systems differently.
  • I4 (answer): Before scoring a reader's answer, a validity check (is it an attempt, is it just the question restated) could short-circuit grading and explain why.
  • I5 (tree): The perceptron parser's feature groups could be ablated per dependency relation to find which families help or harm specific labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud and Houda Bouamor. 2024. QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.28

@inproceedings{bashendy2024qaes,
  title = {{QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays}},
  author = {May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud and Houda Bouamor},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.28},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.28}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Hikmat A. Abdeljaber. 2021. Automatic arabic short answers scoring using longest common subsequence and arabic wordnet. IEEE Access, 9:76433–76445. ↗
  3. Abdelhamid M. Ahmed, Xiao Zhang, Lameya M. Rezk, and Wajdi Zaghouani. 2024. Building an Annotated L1 Arabic/L2 English Bilingual Writer Corpus: The Qatari Corpus of Argumentative Writing (QCAW). Corpus-based Studies across Humanities, 1(1):183– 215. ↗
  4. Mansour Alghamdi, Mohamed Alkanhal, Mohamed Al-Badrashiny, Abdulaziz Al-Qabbany, Ali Areshey, and Abdulaziz Alharbi. 2014. A hybrid automatic scoring system for Arabic essays. AI Communications, 27(2):103–111. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Aqil M. Azmi, Maram F. Al-Jouie, and Muhammad Hussain. 2019. Aaee – automated evaluation of students’ essays in arabic language. Information Processing Management, 56(5):1736–1752. ↗
  7. Domenic V Cicchetti. 1981. Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Applied psychological measurement, 5(1):101– 104. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Afrizal Doewes, Nughthoh Kurdhi, and Akrati Saxena. 2023a. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In 16th International Conference on Educational Data Mining, EDM 2023, pages 103– 113. International Educational Data Mining Society (IEDMS). find
  10. Afrizal Doewes, Nughthoh Arfawi Kurdhi, and Akrati Saxena. 2023b. Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring. In Proceedings of the 16th International Conference on Educational Data Mining, pages 103–113. International Educational Data Mining Society. find
  11. A. Esuli, S. Baccianella, and F. Sebastiani. 2009. Evaluation measures for ordinal regression. In Intelligent Systems Design and Applications, International Conference on, pages 283–287, Los Alamitos, CA, USA. IEEE Computer Society. find
  12. Marwa M. Gaheen, Rania M. ElEraky, and Ahmed A. Ewees. 2020. Automated students arabic essay scoring using trained neural network by e-jaya optimization to support personalized system of instruction. Education and Information Technologies, 26(1):1165–1181. ↗
  13. Wael Hassan Gomaa. 2014. Arabic Short Answer Scoring with Effective Feedback for Students Arabic Short Answer Scoring with Effective Feedback for Students. (January). find
  14. Wael Hassan Gomaa and Aly Aly Fahmy. 2014. Automatic scoring for answers to arabic test questions. Computer Speech Language, 28(4):833–857. ↗
  15. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  16. Beata Beigman Klebanov and Nitin Madnani. 2022. Automated essay scoring. Springer Nature. ↗
  17. Rahul Kumar, Sandeep Mathias, Sriparna Saha, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1485–1495, Seattle, United States. Association for Computational Linguistics. ↗
  18. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  19. Omar Nael, Youssef ELmanyalawy, and Nada Sharaf. 2022. Arascore: A deep learning-based system for arabic short answer scoring. Array, 13:100109. ↗
  20. Christopher Michael Ormerod. 2022. Mapping between hidden states and features to validate automated essay scoring using deberta models. Psychological Test and Assessment Modeling, 64(4):495–526. find
  21. Leila Ouahrani and Djamal Bennouar. 2020. AR-ASAG an ARabic dataset for automatic short answer grading evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2634–2643, Marseille, France. European Language Resources Association.
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  23. Abdulaziz Shehab, Mahmoud Faroun, and Magdi Rashad. 2018. An automatic arabic essay grading system based on text similarity algorithms. International Journal of Advanced Computer Science and Applications, 9. ↗
  24. David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1):2–13. ↗
  25. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724–2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  26. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bashendy-2024-qaes-first-publicly-available-trait/