Elamur Research⌕
☰
home / papers / 2024 / Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system

Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system

Summary

Research on scoring Arabic essays automatically has been held back because no public collection of full-length Arabic essays with grades existed. Earlier work mostly covered one- or two-sentence answers and used hand-built features with shallow classifiers. The authors built AR-AES from undergraduate exams at Umm Al-Qura University. It contains 2046 typed essays answering 12 questions across four courses, from both in-person and online exams. Each essay carries two human marks, the student's gender, the grading rubric and a reference answer. They fine-tuned AraBERT as a score classifier and evaluated it at several levels: the whole dataset, each course, each question, each essay type, each exam mode and each gender. Beyond agreement metrics, they also measured how far wrong predictions land from the first marker's grade. The model was usually exact or off by one point. Two human markers agreed far less with each other than the model agreed with the first marker. This suggests that automatic scoring could help keep grading consistent across large classes.

Key points

  • Dataset: AR-AES contains 2046 essays across 12 questions, totalling 115,454 tokens with 12,440 distinct tokens.
    • The four courses span Biotechnology, Environmental Chemistry, Management Information Systems and Introduction to Information Science.
    • Essay types are argumentative, narrative and source-dependent.
  • Best result: AraBERT reached a QWK of 0.971 on Environmental Chemistry when trained on several questions from that course.
    • Narrative questions, the most open-ended, gave the lowest QWK.
  • Model error size (grades 0–5): 79.49% of predictions exactly matched the first marker, and about 95–96% were within one point.
    • Most errors were overestimates: about 10% were one point too high and 6% one point too low.
    • Source-dependent essays matched exactly in 87% of cases.
  • Human agreement: the most consistent question was Q3 (source-dependent), where the two markers gave the same grade only 30.3% of the time.
    • For 37.1% of Q3 answers, the two markers differed by more than one point.
    • Second markers tended to grade more strictly than the course directors.
  • Other contrasts: the model scored better on online exams than on traditional ones.
    • Within one course, it predicted female students' marks better than male students' marks; the two groups are taught separately by different lecturers.

Interesting ideas

  • I1. Instead of reporting only a single agreement number like QWK, plot the signed difference between prediction and gold grade, since a two-point miss harms a student far more than a one-point miss. — For Mova (answer): grade the school-test and word-problem readers by the distribution of how wrong they are, so near-misses can be told apart from badly broken answers.
  • I2. Measure agreement between human annotators on the same items and use it as the realistic ceiling for judging the model. — For Mova (learn): compute how often the witness parsers or LLM teacher tiers agree with each other, then judge silver-trained models against that disagreement level rather than against an imagined perfect gold.
  • I3. Break results down by how constrained the task is (source-dependent versus narrative); the paper shows performance falls as answers become more open-ended. — For Mova (answer): tag QA items as text-bound versus free-form, so weaknesses in world-building show up separately from the inherent vagueness of open questions.
  • I4. Release the rubric and a model answer with every item, so scores can be traced back to explicit criteria. — For Mova (answer): store a reference answer and checklist for each test question, so the reader's grade can be explained criterion by criterion instead of given as an opaque score.
  • I5. Track the direction of disagreement as well as its size: the model tends to overestimate, while second human markers lean harsher. — For Mova (learn): record whether repair rules or teacher tiers consistently over- or under-attach a label, so systematic bias can be corrected rather than averaged away.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.11212 · arXiv 2407.11212

@misc{ghazawi2024automated,
  title = {{Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system}},
  author = {Ghazawi, Rayed and Simpson, Edwin},
  year = {2024},
  doi = {10.48550/arxiv.2407.11212},
  eprint = {2407.11212},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.11212}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aini, Q., Julianto, A.E., Purbohadi, D.: Development of a scoring application for indonesian language essay questions pp. 6–10 (2018) find
  2. Al-Shalabi, E.F.: An Automated System for Essay Scoring of Online Exams in Arabic based on Stemming Techniques and Levenshtein Edit Operations. International Journal of Computer Science Issues 13(5), 45–50 (9 2016) find
  3. Al-Shargabi, B., Alzyadat, R., Hamad, F.: AEGD: ARABIC ESSAY GRADING DATASET FOR MACHINE LEARNING A Cultural E-Government Readiness Model View project A comparative study for Arabic text classification algorithms based on stop words elimination View project (2021) find
  4. Alduailej, A., Alothaim, A.: AraXLNet: pre-trained language model for sentiment analysis of Arabic. Journal of Big Data 9(1) (12 2022) find
  5. Alghamdi, M., Alkanhal, M., Al-Badrashiny, M., Al-Qabbany, A., Areshey, A., Alharbi, A.: A hybrid automatic scoring system for Arabic essays. AI Communications 27(2), 103–111 (2014). https://doi.org/10.3233/AIC-130586 ↗
  6. Alqahtani, A., Alsaif, A.: Automatic Evaluation for Arabic Essays: A Rule-Based System (2019) find
  7. Alqahtani, A., Alsaif, A.: Automated Arabic essay evaluation. In: Proceedings of the 17th International Conference on Natural Language Processing (ICON). pp. 181–190. NLP Association of India (NLPAI), Indian Institute of Technology Patna, Patna, India (Dec 2020), https://aclanthology.org/2020.icon-main.24 ↗
  8. Alwakid, G., Osman, T., Hughes-Roberts, T.: Challenges in sentiment analysis for Arabic social networks. In: Procedia Computer Science. vol. 117, pp. 89–100. Elsevier B.V. (2017). https://doi.org/10.1016/j.procs.2017.10.097 ↗
  9. Antoun, W., Baly, F., Hajj, H.: AraBERT: Transformer-based Model for Arabic Language Understanding (2 2020), http://arxiv.org/abs/2003. 00104 ↗ our card
  10. Antoun, W., Baly, F., Hajj, H.: Aragpt2: Pre-trained transformer for arabic language generation. arXiv preprint arXiv:2012.15520 (2020) ↗
  11. Ashburn, R.: An experiment in the essay-type question. The Journal of Experimental Education 7(1), 1–3 (1938) find
  12. Awwad, A.M., Schindler, C., Luhana, K.K., Ali, Z., Spieler, B.: Improving pocket paint usability via material design compliance and internationalization & localization support on application level. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI 2017. Association for Computing Machinery, Inc (9 2017) find
  13. Azmi, A.M., Al-Jouie, M.F., Hussain, M.: AAEE – Automated evaluation of students’ essays in Arabic language. Information Processing and Management 56(5), 1736–1752 (9 2019). https://doi.org/10.1016/j.ipm.2019.05.008 ↗
  14. Cohen, J.: Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70(4), 213 (1968) find
  15. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (10 2018) find
  16. Elkateb, S., Black, W., Rodríguez, H., Alkhalifa, M., Vossen, P., Pease, A., Fellbaum, C.: Building a WordNet for Arabic (2006) find
  17. Gaheen, M.M., ElEraky, R.M., Ewees, A.A.: Automated students Arabic essay scoring using trained neural network by e-jaya optimization to support personalized system of instruction. Education and Information Technologies 26(1), 1165–1181 (1 2021). https://doi.org/10.1007/s10639-020-10300-6 ↗
  18. Gong, J., Hu, X., Song, W., Fu, R., Sheng, Z., Zhu, B., Wang, S., ¶£, �., Liu, T.: IFlyEA: A Chinese Essay Assessment System with Automated Rating, Review Generation, and Recommendation pp. 240–248 (2021) find
  19. Hamdi, A., Shaban, K., Zainal, A.: A review on challenging issues in Arabic sentiment analysis 12(9), 471–481 (2016). https://doi.org/10.3844/jcssp.2016.471.481 ↗
  20. Ilgaz, H., Adanır, G.A.: Providing online exams for online learners: Does it really matter for them? Education and Information Technologies 25, 1255–1269 (3 2020) find
  21. Isleem, M.: Developing attitudes toward learning Arabic as a foreign language among american university andcollege students (2014) find
  22. Johnson, M.J.: Gender differences in writing self-beliefs of elementary school students (1999) find
  23. Kaddoura, S., D. Ahmed, R., Jude Hemanth, D.: A comprehensive review on Arabic word sense disambiguation for natural language processing applications 12(4) (7 2022). https://doi.org/10.1002/widm.1447 ↗
  24. Kanan, T., Sadaqa, O., Aldajeh, A., Alshwabka, H., AlZu’bi, S., Elbes, M., Hawashin, B., Alia, M.A., others: A Review of Natural Language Processing and Machine Learning Tools Used to Analyze Arabic Social Media (2019) find
  25. Ke, Z., Ng, V.: Automated essay scoring: A survey of the state of the art. In: IJCAI. vol. 19, pp. 6300–6308 (2019) find
  26. Lea, M.R., Street, B.V.: Student writing in higher education: An academic literacies approach. Studies in Higher Education 23, 157–172 (1998) find
  27. Mahmoud, A., Zrigui, M.: Sentence embedding and convolutional neural network for semantic textual similarity detection in arabic language. Arabian Journal for Science and Engineering 44, 9263–9274 (2019) find
  28. Mathias, S., Bhattacharyya, P.: Can neural networks automatically score essay traits? In: Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. pp. 85–91. Association for Computational Linguistics, Seattle, WA, USA → Online (Jul 2020) find
  29. Meccawy, M., Bayazed, A.A., Al-Abdullah, B., Algamdi, H.: Automatic essay scoring for Arabic short answer questions using text mining techniques. International Journal of Advanced Computer Science and Applications 14(6) (2023) find
  30. Miaschi, A., Brunato, D., Dell’orletta, F., Venturi, G.: What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity pp. 40–47 (2021), www.italianlp.it find
  31. Omar, A., Aldawsari, M.: Lexical Ambiguity in Arabic Information Retrieval: The Case of Six Web-Based Search Engines. International Journal of English Linguistics 10(3), 219 (4 2020). https://doi.org/10.5539/ijel.v10n3p219 ↗
  32. Page, E.B.: The Imminence of... Grading Essays by Computer 47(5), 238– 243 (1966), https://www.jstor.org/stable/20371545 ↗
  33. Phandi, P., Chai, K.M.A., Ng, H.T.: Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression pp. 17–21 (2015) find
  34. Smith, D., Campbell, J., Brooker, R.: The impact of students’ approaches to essay writing on the quality of their essays. Assessment & Evaluation in Higher Education 24(3), 327–338 (1999) find
  35. Soliman, A.B., Eissa, K., El-Beltagy, S.R.: Aravec: A set of Arabic word embedding models for use in arabic nlp. Procedia Computer Science 117, 256–265 (2017) find
  36. Soudi, A., Neumann, G., Bosch, A.v.d.: Arabic Computational Morphology: Knowledge-Basedand Empirical Methods. Computational Linguistics 34(3), 459–462 (9 2008). https://doi.org/10.1162/coli.2008.34.3.459 ↗
  37. Taghipour, K., Tou Ng, H.: A Neural Approach to Automated Essay Scoring pp. 1882–1891 (2016), https://www.kaggle.com/c/asap-aes ↗ our card
  38. Wang, Y., Wang, C., Li, R., Lin, H.: On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation (5 2022) find
  39. Zhu, W.: Faculty views on the importance of writing, the nature of academic writing, and teaching and responding to writing in the disciplines. Journal of Second Language Writing 13, 29–48 (5 2004). https://doi.org/10.1016/j.jslw.2004.04.004 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghazawi-2024-automated-essay-scoring-arabic-dataset/