Elamur Research⌕
☰
home / papers / 2020 / Automated Arabic Essay Evaluation

Automated Arabic Essay Evaluation

Paper · 2020 · Abeer Alqahtani and Amal Alsaif

Summary

The authors build a system that grades Modern Standard Arabic essays automatically. Unlike earlier Arabic work, it needs no reference or model essay from the same topic. Earlier Arabic systems mostly scored short answers against an instructor's answer, or used LSA, which needs topic-matched training essays. The authors' own earlier rule-based scorer had no semantic component.

The new system trains a separate support vector regression (SVR) model for each of five criteria: spelling, structure, coherence, style and punctuation. The holistic grade is then assembled from these five predictions. Each model draws on a large pool of numeric features at several levels: surface, morphological, syntactic, semantic and discourse.

The data are 200 essays by native speakers: university students, plus unedited and published newspaper pieces. Two Arabic-language experts scored them with a fixed rubric.

Overall, 96% of essays fall within a 25% tolerance of the expert score, and the correlation with the experts is 0.87. Both figures exceed previously reported Arabic results. Per-criterion scores also come out, which makes targeted feedback possible.

Key points

  • Data: 200 MSA essays averaging about 250 words, from three sources: students, unedited newspaper submissions and published articles. Two annotators held master's degrees in Arabic and followed the rubric of Alqahtani & Alsaif (2019).
  • Model and tools: SVR, with one model per criterion. Preprocessing used normalization and the Buckwalter stemmer, and FARASA output fed the spelling-error features. Spelling errors fall into four classes: hamza errors, replaced letters, extra letters and missing letters.
  • Overall accuracy: 90% at a tolerance of t = 17% and 96% at t = 25%, with a correlation of 0.87.
  • Per-criterion accuracy: spelling 77%, structure 91%, coherence 87%, style 78%, punctuation 93%.
  • Comparison: prior Arabic systems reported correlations of 0.78 (Alghamdi et al., 2014) and 0.756 (Azmi et al., 2019), and the earlier rule-based system reached 73% accuracy. These comparisons are not direct, because the other datasets are unavailable. Doubling the data from 100 to 200 essays improved results.

Interesting ideas

  • I1. Scoring each rubric criterion with its own model and then combining the outputs keeps the final grade decomposable: each part can be traced and explained to the learner.
  • I2. The features are computed separately for the first, second, middle and last paragraphs, so position helps reveal a segment's role. For example, a first paragraph of ten words or fewer is probably a title.
  • I3. When writers omit punctuation, discourse connectives serve as substitute clause boundaries. One of the strongest punctuation features counts punctuation marks that are not followed by a connective or conjunction.
  • I4. Accuracy is reported at explicit tolerance thresholds (17% and 25%) alongside correlation. This shows how the verdict on the system shifts with the acceptable error margin.
  • I5. A count of synonyms across the whole essay acts as a style signal, rewarding writers who vary their wording instead of repeating the same words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Graders of school-course answers could report separate sub-scores, such as content correctness, reasoning steps and form, instead of a single opaque grade.
  • I2 (world): Document-level structure (title, introduction, body, conclusion) could be inferred from cheap positional cues before deeper analysis assigns discourse roles to the parts of a text.
  • I3 (tokens): A sentence splitter could fall back on connective lists when punctuation is sparse or unreliable, as in learner or informal text.
  • I4 (learn): Evaluations of graded or numeric outputs could always state the tolerance used and report several levels, so improvements are not overstated.
  • I5 (back): The generator could check its own output for unvaried lexical repetition, using WordNet synonym sets to measure and improve variety.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Islamic University.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksEducation · 203 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Latent semantic indexing · 29 works

Cite

Cite as Abeer Alqahtani and Amal Alsaif. 2020. Automated Arabic Essay Evaluation. Proceedings of the 17th International Conference on Natural Language Processing (ICON).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alqahtani2020automated,
  title = {{Automated Arabic Essay Evaluation}},
  author = {Abeer Alqahtani and Amal Alsaif},
  booktitle = {Proceedings of the 17th International Conference on Natural Language Processing (ICON)},
  year = {2020},
  url = {https://aclanthology.org/2020.icon-main.24/}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bakr, A., Mohammad, S., Eissa, K., & El-Beltagy, S. R. (2017). AraVec : A set of Arabic Word Embedding Models for use in Arabic NLP ScienceDirect (November). https://doi.org/10.1016/j.procs.2017.10.117 ↗
  2. Al-Shalabi, E. F. (2016). An automated system for essay scoring of online exams in Arabic based on stemming techniques and Levenshtein edit operations. International Journal of Computer Science Issues, 13(5), 45–50. ↗
  3. Benesty, J., Chen, J., Huang, Y., & Cohen, I. (2009). Pearson Correlation Coefficient. In Noise Reduction in Speech Processing (pp. 1–4). Springer: Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-00296-0_5 ↗
  4. Alghamdi, M., Alkanhal, M., Al-Badrashiny, M., Al-Qabbany, A., Areshey, A., & Alharbi, A. (2014). A hybrid automatic scoring system for Arabic essays. AI Communications, 27(2), 103–111. https://doi.org/10.3233/AIC-130586 ↗
  5. Buckwalter, T. (2004). Buckwalter Arabic morphological analyzer version 2.0. find
  6. Linguistic Data Ali, K. (2018). Inna and Its Sisters. Retrieved from https://arabicblog.info/inna-and-its-sisters ↗
  7. Consortium, University of Pennsylvania, 2002. LDC catalog no.: LDC2004l02. Technical report, ISBN 1-58563-324-0. find
  8. Pasha, A., Al-Badrashiny, M., Diab, M., Kholy, A. El, Eskander, R., Habash, N., … Roth, R. M. (2014). MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. Proceedings of the 9th Language Resources and Evaluation Conference (LREC’14), 1094– 1101.
  9. Burstein, J. (2003). The e-rater scoring engine: Automated essay scoring with natural language processing. Automated Essay Scoring: A Cross-Disciplinary Perspective, 113–121. find
  10. SAFAR: Software Architecture For ARabic. (2013). Retrieved from http://arabic.emi.ac.ma/safar/ ?q=examples# Chung, G., & O’Neil, G. (1997). Methodological Approaches to Online Scoring of Essays, CSE Technical Report 461, University of Southern California CRESST, December, Center for the Study of Evaluation, CRESST, 1522(310). ↗
  11. Shehab, A., Faroun, M., & Rashad, M. (2018). An Automatic Arabic Essay Grading System based on Text Similarity Algorithms, (April). https://doi.org/10.14569/IJACSA.2018.090337. ↗
  12. Dikli, S. (2006). An Overview of Automated Scoring of Essays. Journal Of Technology Learning And Assessment, 5(1), 2006–12. Retrieved from http://www.jtla.org ↗
  13. Surya, D., Madala, V., Krishna, S., Surya, D., Madala, V., Gangal, A., … Sureka, A. (2018). An empirical analysis of machine learning models for automated essay grading. ↗
  14. Dong, F., & Zhang, Y. (2016). Automatic Features for Essay Scoring—An Empirical Study. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1966, 1072–1077. https://doi.org/10.18653/v1/D16-1115 ↗
  15. Witten, I. H., E. Frank, L. E. Trigg, M. A. Hall, G. Holmes and S. J. Cunningham. (1999). Weka: Practical machine learning tools and techniques with Java implementations. Proc ICONIP/ ANZIIS/ANNES99 Future Directions for Intelligent Systems and Information Sciences, pp. 192–196. find
  16. Elliot, S. (2003). IntelliMetric: From here to validity. in M. Shermis and J. Burstein (eds.) Automated Essay Scoring: A Cross-Disciplinary Perspective, Routledge, 71–86. DOI: https://doi.org/ 10.4324/9781410606860 ↗
  17. Ying, X. (2019). An Overview of Overfitting and its Solutions: IOP Conf. Series: Journal of Physics: Conf. Series 1168. 2019. find
  18. FARASA: Advanced Tools for Arabic. (2019). Retrieved from http://qatsdemo.cloudapp.net/farasa/ ↗
  19. Gomaa, W. H., & Fahmy, A. A. (2014). Automatic scoring for answers to Arabic test questions. Computer Speech and Language, 28(4), 833–857. https://doi.org/10.1016/j.csl.2013.10.005 ↗
  20. Ibrahim, M. (2006). Looks at the technical article in modern Arabic literature ( بدﻷا ﻲﻓ ﻲﻨﻔﻟا لﺎﻘﻤﻟا ﻲﻓ تاﺮﻈﻧ ﺚﯾﺪﺤﻟا ﻲﺑﺮﻌﻟا). pp.21–22. find
  21. Janda, H. K., Pawar, A., Du, S., & Mago, V. (2019). Syntactic, Semantic and Sentiment Analysis: The Joint Effect on Automated Essay Evaluation. IEEE Access, 7, 108486–108503. ↗
  22. Kukich, K. (2000). Beyond automated essay Scoring. IEEE Intelligent Systems and Their Applications. https://doi.org/10.1109/5254.889104 ↗
  23. Lemaire, B., & Dessus, P. (2001). A System to Assess the Semantic Content of Student Essays. Journal of Educational Computing Research, 24(3), 305–320. https://doi.org/10.2190/G649-0R9C-C021-P6X3 ↗
  24. Nahar, I. K. M., & Alsmadi, I. M. (2009). The Automatic Grading for Online exams in Arabic with Essay Questions Using Statistical and computational linguistics Techniques, 1(2), 215– 220. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alqahtani-2020-automated-arabic-essay-evaluation/