Elamur Research⌕
☰
home / papers / 2020 / AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation

AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation

Summary

Automatic grading of short free-text answers needs gold-standard test collections, and before this work Arabic had none that were publicly available. The authors build AR-ASAG from real exams in a cybercrime course taught in Arabic to master students. Each student answer is paired with a teacher's model answer and scored by two human experts. They also propose an unsupervised grader for low-resource settings, which needs only a plain text corpus and a stemmer. The grader learns word vectors with COALS, a co-occurrence-based distributional method. It builds answer vectors by summing word vectors and adds term weighting and a shared-word component, then compares the student answer with the model answer. Experiments vary three factors: how domain-specific the training corpus is, the dimension of the semantic space, and the stemming technique. The approach is tested on AR-ASAG and on the Arabic-Arabic STS 250 set from SemEval-2017. The authors call the results promising and intend them as a baseline. The resource matters because earlier Arabic grading work used private or ad hoc examples, so methods could not be compared.

Key points

  • AR-ASAG has 2133 (Model Answer, Student Answer) pairs and is distributed as txt, xml, Moodle xml and .db (ISLRN 529-005-230-448-6). A companion Arabic Cyber Text Corpus is also released.
  • The data comes from three exams of 16 questions each (48 questions in total), taken under normal exam conditions by about 170 native Arabic-speaking students. Duplicate answers to a question are kept only once.
  • There are five question types: define, explain, what consequences, justify, and what is the difference. Each question has one model answer.
  • Two computer science teachers graded every answer from 0 to 5. The average of their two grades is the gold score.
  • For comparison, the only earlier Arabic dataset (Cairo University) has 610 answers and is not public. The English Texas dataset has 630 answers. Evaluation also uses STS 250 SemEval-2017 track 1.

Interesting ideas

  • I1. Identical student answers are stored only once, so frequent stock answers cannot inflate agreement scores or bias the evaluation toward easy cases. — For Mova (learn): Before computing accuracy, deduplicate test and silver items by their normalized surface form, so that repeated easy sentences do not mask failures on rarer ones.
  • I2. Grading is framed as comparing a student answer with a single teacher reference answer, so no per-question training or hand-written patterns are needed. — For Mova (answer): A student reader could check an answer by aligning its frames and world entities with those of a reference answer, producing an explainable partial score instead of right/wrong.
  • I3. The gold score averages two independent annotators, and related datasets report the correlation between human graders. That human-human agreement gives a natural ceiling for any automatic grader. — For Mova (answer): Report Mova's school-test scores next to inter-annotator agreement, so a gap shows how much is left to gain and whether the gold itself is noisy.
  • I4. The experiments vary one language-dependent component at a time (stemming technique, corpus domain, space dimension) to see how much each affects final grading quality. — For Mova (tokens): Run the same downstream QA with lemmas, crude stems and raw forms to measure how much the morphology crate actually contributes.
  • I5. The dataset labels each question by type (define, explain, consequences, justify, difference), so results can be broken down by type of reasoning required. — For Mova (answer): Tag Mova's test questions by type (definition, cause, contrast, consequence) and report accuracy per type, so it is clear which reasoning paths in the world model are weak.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ouahrani2020ar,
  title = {{AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation}},
  author = {Ouahrani, Leila and Bennouar, Djamal},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.321/}
}

References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Thubaity, A. O. (2015). A 700M+ Arabic corpus: KACST Arabic corpus design and construction. Language Resources and Evaluation, 49(3), 721– 751. https://doi.org/10.1007/s10579-014-9284-1 ↗
  2. Anderson, Lorin W.; Krathwohl, David R.; Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing : a revision of Bloom’s taxonomy of educational objectives (omplete ed., Ed.). New York : Longman. find
  3. Bennouar, D. (2017). An Automatic Grading System Based on Dynamic Corpora. The International Arab Journal of Information Technology, 14(4A). find
  4. Brown, Sally, Ed.; Glasner, Angela, E. (1999). Assessment Matters in Higher Education: Choosing and Using Diverse Approaches., 1999. find
  5. Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. In International Journal of Artificial Intelligence in Education (Vol. 25). https://doi.org/10.1007/s40593-014-0026-8 ↗
  6. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017). SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation. In Association for Computational Linguistics (Ed.), Proceedings ofthe 11th International Workshop on Semantic Evaluations (SemEval-2017) (pp. 1–14). https://doi.org/10.18653/v1/S17-2001 ↗
  7. El Moatez Billah Nagoudi, Jérémy Ferrero, D. S. (2017). LIM-LIG at SemEval-2017 Task1: Enhancing the Semantic Similarity for Arabic Sentences with Vectors Weighting. In Association for Computational Linguistics (Ed.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) (pp. 134–138). find
  8. Gomaa, W. H., & Fahmy, A. A. (2014a). Arabic Short Answer Scoring with Effective Feedback for Students. International Journal of Computer Applications, 86(2), 35–41. https://doi.org/10.5120/14961-3177 ↗
  9. Gomaa, W. H., & Fahmy, A. A. (2014b). Automatic scoring for answers to Arabic test questions. Computer Speech and Language, 28(4), 833–857. https://doi.org/10.1016/j.csl.2013.10.005 ↗
  10. Gratta, R. del, Frontini, F., Fahad, K., Mariani, J., & Soria, C. (2014). The LREMap for under-resourced languages. find
  11. J. B. MacQueen. (1967). Some Methods for classification and Analysis of Multivariate Observations. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability, 281–297. find
  12. Khoja, S., & Garside, R. (1999). Stemming arabic text. Lancaster, UK, Computing Department, Lancaster University. find
  13. LEE R. DICE. (2012). Measures of the Amount of Ecologic Association Between Species Author ( s ): Lee R . Dice Reviewed work ( s ): Published by : Ecological Society of America Stable URL : http://www.jstor.org/stable/1932409 . 26(3), 297– 302. ↗
  14. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C. (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice. https://doi.org/10.1111/emip.12028 ↗
  15. Lovins, J. (1963). Development of a stemming algorithm. Cambridge: M.I.T. Information Processing Group Electronic Systems Laboratory. find
  16. Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. https://doi.org/10.3758/BF03204766 ↗
  17. Mag`ooda, A., Zahran, M. A., Rashwan, M., Raafat, H., & Fayek, M. B. (2016). Vector Based Techniques for Short Answer Grading. FLAIRS Conference, 238– 243. find
  18. Mahmoud El-Haj, Udo Kruschwitz, Colchester, & Fox, C. (2015). Creating Language Resources for Under-resourced Languages : Methodologies , and experiments with Arabic. Language Resources and Evaluation, 49(3), 549–580. https://doi.org/10.1007/s10579-014-9274-3 ↗
  19. Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. Proceedings of the 21st National Conference on Artificial Intelligence, 1, 775–780. https://doi.org/10.1.1.65.3690 ↗
  20. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Retrieved from http://arxiv.org/abs/1301.3781 ↗
  21. Mohler, M., Bunescu, R., & Mihalcea, R. (2011). Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 752–762. find
  22. Mohler, M., & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL ’09, 567–575. https://doi.org/10.3115/1609067.1609130 ↗
  23. Rohde, D. L. T., Gonnerman, L. M., & Plaut, D. C. (2004). An Improved Method for Deriving Word Meaning from Lexical. Cognitive Psychology, 7, 573–605. find
  24. Sahlgren, M., (2005). An introduction to random indexing. In Methods and Applications of Semantic Indexing Worshop at the 7th International Conference on Terminology and Knowldge Engineering, TKE 2005. find
  25. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0 ↗
  26. Shourya Roy, Y. Narahari, and O. D. D. (2015). A Perspective on Computer Assisted Assessment Techniques for Short Free-text Answers. Communications in Computer and Information Science, 571(June 2015). https://doi.org/10.1007/978-3-319-27704-2 ↗
  27. Wu, H., Huang, H., Jian, P., Guo, Y., & Su, C. (2017). BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity. In Association for Computational Linguistics (Ed.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017): Vol. L (pp. 77–84). find
  28. Zahran, M. A., Magooda, A., Mahgoub, A. Y., Raafat, H., Rashwan, M., & Atyia, A. (2015). Word Representations in Vector Space and their Applications for Arabic. In A. Gelbukh (Ed.) (Ed.), 16th international conference, CICLing 2015 Cairo, Egypt, april 14 (Vol. 9041, pp. 430–443). https://doi.org/10.1007/978-3-319-18111-0_32 9. ↗
  29. Language Resource References (AR-ASAG Dataset, 2020). The Arabic Dataset for Automatic Short Answer Grading Evaluation, V. 1.0, ISLRN 529-005-230-448-6. find
  30. (Arabic Cyber Text Corpus, 2020). The Arabic In-Domain Cyber Text Corpus, V. 1.0, ISLRN 798-080-268-332-8. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ouahrani-2020-ar-asag-arabic-dataset-automatic/