Elamur Research⌕
☰
home / papers / 2025 / Evaluating Prompt Relevance in Arabic Automatic Essay Scoring: Insights from Synthetic and Real-World Data

Evaluating Prompt Relevance in Arabic Automatic Essay Scoring: Insights from Synthetic and Real-World Data

Paper · 2025 · Chatrine Qwaider, Kirill Chirkunov, Bashar Alhafni et al.

Summary

The paper tackles a question Arabic essay scoring had not modelled explicitly: does a student's essay actually address the prompt it was written for? The authors treat this as binary on-topic/off-topic classification. Such a classifier can serve two roles: screening out off-topic essays before human annotation, and acting as a gate in front of a scoring model. Because labelled Arabic data is scarce, the main training set is synthetic: GPT-4o essays, each paired once with its own prompt and once with a randomly chosen foreign prompt. Real-world testing uses QAES plus a filtered ZAEBUC, with negatives built the same way. The authors compare five prompt expansion strategies (AraVec, CAMeLBERT, two POS-guided variants, GPT-4o) and several classifiers: cosine thresholds over many embedding models, SVMs, causal LLMs, and a fine-tuned SBERT. The fine-tuned SBERT does best on synthetic data and stays strong on the mixed real set. It is a much cheaper option than LLMs, and the study gives Arabic its first prompt-relevance benchmark.

Key points

  • Data: QAES has 195 essays on two similar prompts, and its ambiguous "partially relevant" class was dropped. ZAEBUC's 214 essays were reduced to 176 after a GPT check flagged mixed-prompt essays. Together they cover five prompts and 608 pairs. The synthetic set has 3,220 essays on 155 prompts, giving 6,440 pairs, split by prompt so no prompt or essay crosses splits.

  • GPT-4o expansion generates five subheaders per prompt, each with five clue words a student might plausibly use.

  • Fine-tuning paraphrase-multilingual-MiniLM-L12-v2 with a cosine similarity loss gives these F1 scores:

    Test setOriginal promptsGPT-expanded prompts
    Synthetic test97%97%
    QAES+ZAEBUC91%86%
    QAES alone85%76%

    The abstract reports 98% on synthetic data.

  • Raw-similarity effects vary by model. For MiniLM, the unexpanded prompt gives the widest gap between relevant and non-relevant cosine scores (0.40). For distiluse, GPT expansion widens the gap (0.35 vs 0.32).

  • On ZAEBUC, F1 is 95.6% on raw learner essays and 96.8% on their manually corrected versions. Remaining errors trace mainly to structural differences, not vocabulary. Compared with the synthetic texts, real essays pack more words into each sentence (a median of 12 against 8) and each paragraph (96 against 44), and are split into paragraphs less often.

Interesting ideas

  • I1. Off-topic negatives cost nothing: pair each essay with a randomly reassigned prompt, giving perfectly balanced labels without any annotation.
  • I2. Splitting synthetic data by prompt, and building pairs separately inside each split, keeps topical knowledge from leaking between train and test.
  • I3. Running the same model on raw learner text and on its hand-corrected version isolates how much surface noise actually costs.
  • I4. The error analysis measures how far each misclassified sample sits from the synthetic data's structural medians (sentence length, paragraph length, breaks), which shows that structure, not vocabulary, drives the gap.
  • I5. A cheap relevance gate placed before the expensive scorer keeps off-topic inputs from receiving inflated, meaningless scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mismatched text/question pairs could serve as automatic negative controls, letting us check that a reader refuses or flags questions that do not belong to the story.
  • I2 (learn): When generating silver or synthetic training items, split by source document or template rather than by item, so evaluation measures transfer and not memorised topics.
  • I3 (tokens): Paired noisy/corrected inputs would measure how far tagger and lemmatizer errors on misspelled text propagate, separately from genuine domain gaps.
  • I4 (tree): Bucketing gparse or perceptron parse errors by sentence length and layout distance from the treebank's typical shape could reveal where treebank-to-real-text shift actually hurts.
  • I5 (answer): A frame-overlap check between the question and the text's world could decide early whether a question is answerable, instead of forcing a confident wrong answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Chatrine Qwaider, Kirill Chirkunov, Bashar Alhafni, Nizar Habash and Ted Briscoe. 2025. Evaluating Prompt Relevance in Arabic Automatic Essay Scoring: Insights from Synthetic and Real-World Data. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.13

@inproceedings{qwaider2025evaluating,
  title = {{Evaluating Prompt Relevance in Arabic Automatic Essay Scoring: Insights from Synthetic and Real-World Data}},
  author = {Chatrine Qwaider, Kirill Chirkunov, Bashar Alhafni, Nizar Habash and Ted Briscoe},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.13},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.13}
}

References

30 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Abdelhamid M Ahmed, Xiao Zhang, Lameya M Rezk, and Wajdi Zaghouani. 2024. Building an annotated l1 arabic/l2 english bilingual writer corpus: the qatari corpus of argumentative writing (qcaw). Corpus-Based Studies across Humanities, 1(1):183–215. ↗
  3. Salam Albatarni, Sohaila Eltanbouly, and Tamer Elsayed. 2024. Graded relevance scoring of written essays with dense retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1329–1338. find
  4. et al. An Yang. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  6. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud, and Houda Bouamor. 2024. Qaes: First publicly-available trait-specific annotations for automated scoring of arabic essays. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 337–351.
  7. Yuan Chen and Xia Li. 2024. Plaes: Prompt-generalized and level-aware learning framework for cross-prompt automated essay scoring. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12775–12786. find
  8. Ronan Cummins, Helen Yannakoudakis, and Ted Briscoe. 2016. Unsupervised modeling of topical relevance in l2 learner text. In Proceedings of the 11th workshop on innovative use of NLP for building educational applications, pages 95–104. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  10. Falcon-LLM-Team. 2025. Falcon-h1: A family of hybrid-head language models redefining efficiency and performance.
  11. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  12. Rayed Ghazawi and Edwin Simpson. 2025. How well can llms grade essays in arabic? Computers and Education: Artificial Intelligence, 9:100449. ↗
  13. Nizar Habash and David Palfreyman. 2022. Zaebuc: An annotated arabic-english bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88. find
  14. Inception. 2024. Jais family model card. find
  15. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  16. Go Inoue, Salam Khalifa, and Nizar Habash. 2022. Morphosyntactic tagging with pre-trained language models for Arabic and its dialects. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1708–1719, Dublin, Ireland. Association for Computational Linguistics.
  17. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2024. Matryoshka representation learning. Preprint, arXiv:2205.13147. ↗
  18. Shengjie Li and Vincent Ng. 2024. Conundrums in cross-prompt automated essay scoring: Making sense of the state of the art. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 7661–7681. ↗
  19. Nourmeen Lotfy, Abdulaziz Shehab, Mohammed Elho-seny, and Ahmed Abu-Elfetouh. 2023. An enhanced automatic arabic essay scoring system based on machine learning algorithms. CMC-COMPUTERS MA-TERIALS & CONTINUA, 77(1):1227–1249. ↗
  20. Sandeep Mathias and Pushpak Bhattacharyya. 2018. Asap++: Enriching the asap automated essay grading dataset with essay attribute scores. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  21. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations. ↗
  22. Zach Nussbaum and Brandon Duderstadt. 2025. Training sparse mixture of experts text embedding models. Preprint, arXiv:2502.07972. ↗
  23. Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534–1543. find
  24. Chatrine Qwaider, Bashar Alhafni, Kirill Chirkunov, Nizar Habash, and Ted Briscoe. 2025. Enhancing Arabic automated essay scoring with synthetic data and error injection. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 549–563, Vienna, Austria. Association for Computational Linguistics. ↗
  25. Marek Rei and Ronan Cummins. 2016. Sentence similarity measures for fine-grained estimation of topical relevance in learner essays. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 283–288, San Diego, CA. Association for Computational Linguistics. find
  26. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  27. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  28. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  29. Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. 2024. Is cosine-similarity of embeddings really about similarity? In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 887–890. ACM. find
  30. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, and 1 others. 2019. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qwaider-2025-evaluating-prompt-relevance-arabic-automatic/