Elamur Research⌕
☰
home / papers / 2025 / Feature Engineering is not Dead: A Step Towards State of the Art for Arabic Automated Essay Scoring

Feature Engineering is not Dead: A Step Towards State of the Art for Arabic Automated Essay Scoring

Paper · 2025 · Marwan Sayed, Sohaila Eltanbouly, May Bashendy et al.

Summary

Automated essay scoring for Arabic has received far less attention than for English. Most earlier Arabic work trains and tests on essays written for a single prompt, and there was no open, systematic feature inventory. The authors assemble and publicly release 816 handcrafted features for Arabic AES. They draw on three sources: an earlier Arabic AES study, the widely used English feature set of Ridley et al., and the features of a recent English state-of-the-art system. The features are grouped into five families: surface, readability, lexical, semantic and syntactic.

They test the features on 620 Arabic essays that carry both holistic and trait-level scores. The tests run in a cross-prompt setting, which the authors describe as the first such study for Arabic. Features are used both alone and combined with encoder representations, and the results are compared against Arabic-centric LLMs. Feature-driven models match and sometimes beat larger models, setting new reference results. This suggests that explicit linguistic measurement still captures quality signals that model capacity alone does not.

Key points

  • The released resource has 816 features in five categories, each split into subcategories. Examples include character, word, sentence and paragraph surface statistics, dialect features, n-gram and punctuation features, prompt adherence, sentiment, text similarity, POS tags and bigrams, Arabic-specific grammatical features and pronouns.
  • The code and features are available at github.com/Maroibo/AES_features.
  • The data is 620 Arabic essays scored holistically and on separate traits such as relevance and organization. Evaluation is cross-prompt, and features are selected with a correlation-threshold method borrowed from Li and Ng (2024).
  • Readability is covered by Arabic formulas (Heeti, OSMAN) and by English formulas (SMOG, Flesch–Kincaid, Linsear Write) applied to Arabic text. The surviving text indicates the Arabic-specific formulas correlated more strongly with scores.
  • In the correlation analysis, surface features ranked highest and semantic features second. Text similarity was always among the top four subcategories. The relevance trait broke this pattern: semantic features led there, and pronoun features were the strongest single subcategory.

Interesting ideas

  • I1. The authors rank feature subcategories per trait by their maximum absolute correlation with the target, which shows which kinds of evidence matter for which judgement.
  • I2. They count how many sentences drift away from the standard variety and use that share as a quality signal, a newly proposed feature group.
  • I3. Prompt adherence is measured as the max, min and mean similarity between each sentence and the prompt, so both the best-aligned and worst-aligned parts are captured.
  • I4. Hybrid models append explicit engineered features to encoder representations, because the features carry signals the encoder underrepresents.
  • I5. The authors note that the importance of feature categories varies across traits, which suggests separate scoring modules for groups of similar traits instead of one model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Diagnostic signals of errors, such as tag confusions, attachment types and frame gaps, could be grouped into families and ranked per error class to decide where repair work pays off most.
  • I2 (tokens): A per-sentence register or dialect detector could flag nonstandard English input before tagging, so downstream errors are attributed to the input rather than to the parser.
  • I3 (answer): A student answer's frames could be scored against the question's frames in the same max/min/mean way, using explainable frame overlap instead of embeddings.
  • I4 (tree): gparse could receive explicit morphological or clause-structure features alongside its transformer states, making some parsing decisions traceable to named inputs.
  • I5 (answer): Different question types, such as maths word problems or fable QA, could use specialised readers that draw on different evidence families from the world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Marwan Sayed, Sohaila Eltanbouly, May Bashendy and Tamer Elsayed. 2025. Feature Engineering is not Dead: A Step Towards State of the Art for Arabic Automated Essay Scoring. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.19

@inproceedings{sayed2025feature,
  title = {{Feature Engineering is not Dead: A Step Towards State of the Art for Arabic Automated Essay Scoring}},
  author = {Marwan Sayed, Sohaila Eltanbouly, May Bashendy and Tamer Elsayed},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.19},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.19}
}

References

40 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hikmat A Abdeljaber. 2021. Automatic arabic short answers scoring using longest common subsequence and arabic wordnet. IEEE Access, 9:76433–76445. ↗
  2. Abdelhamid M. Ahmed, Xiao Zhang, Lameya M. Rezk, and Wajdi Zaghouani. 2024. Building an Annotated L1 Arabic/L2 English Bilingual Writer Corpus: The Qatari Corpus of Argumentative Writing (QCAW). Corpus-based Studies across Humanities, 1(1):183– 215. ↗
  3. Saeda A Al Awaida, Bassam Al-Shargabi, and Thamer Al-Rousan. 2019. Automated arabic essay grading system based on f-score and arabic worldnet. Jordanian Journal of Computers and Information Technology, 5(3). find
  4. Khalaf Al-Heeti. 1984. Judgment Analysis Technique Applied to Readability Prediction of Arabic Reading Material. Ph.D. thesis, ProQuest Dissertations and Theses. Copyright ProQuest LLC. ProQuest does not claim copyright in the individual underlying works. Last updated 2023-02-19. find
  5. Mohammad Alobed, Abdallah M M Altrad, and Zainab Binti Abu Bakar. 2021a. A comparative analysis of euclidean, jaccard and cosine similarity measure and arabic wordnet for automated arabic essay scoring. In 2021 Fifth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 70–74. find
  6. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud, and Houda Bouamor. 2024. QAES: First publicly-available trait-specific annotations for automated scoring of Arabic essays. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 337–351, Bangkok, Thailand. Association for Computational Linguistics.
  7. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc. find
  8. Leo Breiman. 2001. Random forests. Machine Learning, 45(1):5–32. Accessed: YYYY-MM-DD. find
  9. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), pages 785–794, New York, NY, USA. ACM. ↗
  10. Jacob Cohen. 1968. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. ↗
  11. Heejin Do, Yunsu Kim, and Gary Lee. 2024. Autoregressive score generation for multi-trait essay scoring. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1659–1666, St. Julian’s, Malta. Association for Computational Linguistics. find
  12. Heejin Do, Yunsu Kim, and Gary Geunbae Lee. 2023. Prompt- and trait relation-aware cross-prompt essay trait scoring. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1538–1551, Toronto, Canada. Association for Computational Linguistics. find
  13. Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada. Association for Computational Linguistics.
  14. Mahmoud El-Haj and Paul Rayson. 2016. Osman—a novel arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 250–255. find
  15. Sohaila Eltanbouly, Salam Albatarni, and Tamer Elsayed. 2025. TRATES: Trait-specific rubric-assisted cross-prompt essay scoring. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20528–20543, Vienna, Austria. Association for Computational Linguistics. find
  16. Marwa M Gaheen, Rania M ElEraky, and Ahmed A Ewees. 2020. Optimized neural network-based improved multiverse optimizer algorithm for automated arabic essay scoring. INTERNATIONAL JOUR-NAL OF SCIENTIFIC TECHNOLOGY RESEARCH, 9:238–243. find
  17. Marwa M Gaheen, Rania M ElEraky, and Ahmed A Ewees. 2021. Automated students arabic essay scoring using trained neural network by e-jaya optimization to support personalized system of instruction. Education and Information Technologies, 26:1165– 1181. ↗
  18. Francis Galton. 1886. Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15:246– 263. Accessed: YYYY-MM-DD. ↗
  19. Rayed Ghazawi and Edwin Simpson. 2024. Automated essay scoring in arabic: a dataset and analysis of a bert-based system. arXiv preprint arXiv:2407.11212. ↗
  20. Rayed Ghazawi and Edwin Simpson. 2025. How well can llms grade essays in arabic? arXiv preprint arXiv:2501.16516. ↗
  21. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. ↗
  22. Rahul Kumar, Sandeep Mathias, Sriparna Saha, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1485–1495, Seattle, United States. Association for Computational Linguistics. ↗
  23. Shengjie Li and Vincent Ng. 2024. Conundrums in cross-prompt automated essay scoring: Making sense of the state of the art. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7661– 7681, Bangkok, Thailand. Association for Computational Linguistics. ↗
  24. Julian F Lohmann, Fynn Junge, Jens Möller, Johanna Fleckenstein, Ruth Trüb, Stefan Keller, Thorben Jansen, and Andrea Horbach. 2024. Neural networks or linguistic features?-comparing different machine-learning approaches for automated assessment of text quality traits among l1-and l2-learners’ argumentative essays. International Journal of Artificial Intelligence in Education, pages 1–40. ↗
  25. Rim Aroua Machhout and Chiraz Ben Othmane Zribi. 2024. Enhanced bert approach to score arabic essay’s relevance to the prompt. Communications of the IBIMA, 2024. ↗
  26. Somaia Mahmoud, Emad Nabil, and Marwan Torki. 2024. Automatic scoring of arabic essays: A parameter-efficient approach for grammatical assessment. IEEE Access. ↗
  27. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading, 12(8):639–646. find
  28. Abu Bakr Mohammad, Kareem Eissa, and Samhaa El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  29. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  30. John O’Hayre. 1966. Gobbledygook Has Gotta Go. U.S. Department of the Interior, Bureau of Land Management, Denver, Colorado. A style manual that helped inspire the Plain Language movement. find
  31. Christopher Michael Ormerod. 2022. Mapping between hidden states and features to validate automated essay scoring using deberta models. Psychological Test and Assessment Modeling, 64(4):495–526. find
  32. Robert Ridley, Liang He, Xin-yu Dai, Shujian Huang, and Jiajun Chen. 2021. Automated cross-prompt scoring of essay traits. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13745–13753. find
  33. Robert Ridley, Liang He, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2020. Prompt agnostic essay scorer: a domain generalization approach to cross-prompt automated essay scoring. arXiv preprint arXiv:2008.01441. ↗
  34. Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
  35. Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022. On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3416–3425, Seattle, United States. Association for Computational Linguistics. find
  36. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724–2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  37. Jiangsong Xu, Jian Liu, Mingwei Lin, Jiayin Lin, Shenbao Yu, Liang Zhao, and Jun Shen. 2025. Epcts: Enhanced prompt-aware cross-prompt essay trait scoring. Neurocomputing, 621:129283. ↗
  38. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569, Online. Association for Computational Linguistics. find
  39. Mustafa Zeki, Othman O. Khalifa, and A. W. Naji. 2010. Development of an arabic text-to-speech system. In International Conference on Computer and Communication Engineering (ICCCE’10), pages 1–5. find
  40. Chunyun Zhang, Jiqin Deng, Xiaolin Dong, Hongyan Zhao, Kailin Liu, and Chaoran Cui. 2025. Pairwise dual-level alignment for cross-prompt automated essay scoring. Expert Systems with Applications, 265:125924. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sayed-2025-feature-engineering-is-not-dead/