Elamur Research⌕
☰
home / papers / 2023 / In-Context Meta-Learning vs. Semantic Score-Based Similarity: A Comparative Study in Arabic Short Answer Grading

In-Context Meta-Learning vs. Semantic Score-Based Similarity: A Comparative Study in Arabic Short Answer Grading

Paper · 2023 · Menna Fateen and Tsunenori Mine

Summary

The paper looks at automatic short answer grading (ASAG) for Arabic, where labelled data is scarce. It compares two grading styles. The instance-based style learns from many graded answers. The reference-based style scores an answer by how close it is to a model answer.

The authors use AR-ASAG, an exam set on cybercrime, and group questions into five prompt types. They train one scoring model per type with each of two methods:

  • InCML (in-context meta-learning): an AraBERTv2 model whose input packs the answer, the question, the reference answer and graded examples.
  • SSS (score-based semantic similarity): a SentenceTransformers model.

A small SetFit classifier assigns new questions to a prompt type. Its training data is expanded with examples generated by GPT-3.5.

On unseen answers to known questions, both methods beat a majority-class baseline and sometimes match or beat human raters. Neither method wins consistently. On unseen questions (zero-shot), SSS clearly holds up better. This suggests that reference-based grading suits classrooms with little annotated data.

Key points

  • Data: AR-ASAG contains 2133 answers to 48 questions, drawn from 3 cybercrime exams.
    • Questions fall into five prompt types: define, explain, consequences, justify, compare.
    • Two raters scored each answer on a 0–5 scale. Their agreement was 35%, rising to 56% after scores were discretized into 6 classes.
  • Prompt-type classifier: SetFit with an AraBERT body, trained 4-shot, reached only 0.357 accuracy. With 250 examples per type (GPT-3.5 augmentation, "Aug 3"), accuracy rose to 0.893 and F1 to 0.821.
  • Metrics and baselines: evaluation uses QWK and PTA 0–3 accuracy. The comparison points are a majority-class baseline (MV) and the agreement between the two human raters.
  • Unseen answers: results are close to human level and depend on the prompt type. For example, on P3, InCML 0 scores QWK 0.798 against 0.749 for the human raters.
  • Unseen questions: InCML often collapses to QWK near 0, while SSS stays usable. For example, on P1, SSS 30 scores 0.620 against 0.145 for InCML 0. Code and models are public in the GitHub repository mennafateen/SSS-vs-InCML.

Interesting ideas

  • I1. Instead of training one model per question, the authors train one shared model per question type and route each new question to it with a classifier, which cuts data needs and storage.
  • I2. Comparing an answer against a reference answer generalized to new questions much better than learning from previously graded answers.
  • I3. A tiny few-shot classifier jumped from 0.357 to 0.893 accuracy once an LLM generated extra look-alike examples for each class.
  • I4. Noisy, fine-grained ratings from several raters were turned into coarser classes (round and take the median), which raised inter-rater agreement before training.
  • I5. The evaluation keeps "unseen answers" and "unseen questions" as separate splits and reports human agreement as a ceiling next to a majority-class floor.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A first step could tag school test items as define, explain, compare or justify, and then pick a dedicated reader strategy for each type.
  • I2 (answer): Students' answers could be compared with the reference answer at the level of frames and world entities rather than surface text, which keeps grading explainable and usable zero-shot.
  • I3 (learn): An LLM teacher tier could generate synthetic examples for rare constructions or question types. A held-out set of real examples should serve as a negative control to check that the gain is genuine.
  • I4 (learn): When several witness parsers or teachers disagree on graded labels, coarsening the labels before voting can produce cleaner silver data.
  • I5 (learn): Test suites could split by new items versus new item types and always report floor and ceiling controls, which exposes models that only memorize known tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kyushu UniversityFukuoka · Japan4 works · 25 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Menna Fateen and Tsunenori Mine. 2023. In-Context Meta-Learning vs. Semantic Score-Based Similarity: A Comparative Study in Arabic Short Answer Grading. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.28

@inproceedings{fateen2023context,
  title = {{In-Context Meta-Learning vs. Semantic Score-Based Similarity: A Comparative Study in Arabic Short Answer Grading}},
  author = {Menna Fateen and Tsunenori Mine},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.28},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.28}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  2. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2022. Similarity-based content scoring - how to make sbert keep up with bert. In Proceedings of The 17th Workshop on Innovative Use of NLP for Building Educational Applications. find
  3. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2023. Similarity-based content scoring-a more classroom-suitable alternative to instance-based scoring? In Findings of the Association for Computational Linguistics: ACL 2023, pages 1892–1903. find
  4. Aubrey Condor, Max Litster, and Zachary Pardos. 2021. Automatic short answer grading with sbert on out-of-sample questions. International Educational Data Mining Society. find
  5. Nigel Fernandez, Aritra Ghosh, Naiming Liu, Zichao Wang, Benoît Choffin, Richard Baraniuk, and Andrew Lan. 2022. Automated scoring for reading comprehension via in-context bert tuning. In International Conference on Artificial Intelligence in Education, pages 691–697. Springer. find
  6. Wael Hassan Gomaa and Aly Aly Fahmy. 2014. Automatic scoring for answers to arabic test questions. Computer Speech & Language, 28(4):833–857. ↗
  7. Andrea Horbach and Torsten Zesch. 2019. The influence of variance in learner answers on automatic content scoring. In Frontiers in Education, volume 4, page 28. Frontiers Media SA. ↗
  8. John M Keller. 1983. Motivational design of instruction. Instructional design theories and models: An overview of their current status, 1(1983):383–434. find
  9. Jiaqi Lun, Jia Zhu, Yong Tang, and Min Yang. 2020. Multiple data augmentation strategies for improving performance on automatic short answer scoring. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13389–13396. find
  10. Gaoyan Lv, Wei Song, Miaomiao Cheng, and Lizhen Liu. 2021. Exploring the effectiveness of question for neural short answer scoring system. In 2021 IEEE 11th International Conference on Electronics Information and Emergency Communication (ICEIEC) 2021 IEEE 11th International Conference on Electronics Information and Emergency Communication (ICEIEC), pages 1–4. IEEE. find
  11. Maram Meccawy, Afnan Ali Bayazed, Bashayer Al-Abdullah, and Hind Algamdi. 2023. Automatic essay scoring for arabic short answer questions using text mining techniques. International Journal of Advanced Computer Science and Applications, 14(6). ↗
  12. Omar Nael, Youssef ELmanyalawy, and Nada Sharaf. 2022. Arascore: a deep learning-based system for arabic short answer scoring. Array, 13:100109. ↗
  13. Leila Ouahrani and Djamal Bennouar. 2020. Ar-asag an arabic dataset for automatic short answer grading evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2634– 2643.
  14. Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. ↗
  16. Abdulaziz Shehab, Mahmoud Faroun, and Magdi Rashad. 2018. An automatic arabic essay grading system based on text similarity algorithms. International Journal of Advanced Computer Science and Applications, 9(3). ↗
  17. Gary F Simons, Abbey L Thomas, and Chad K White. 2022. Assessing digital language support on a global scale. arXiv preprint arXiv:2209.13515. ↗
  18. Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. arXiv preprint arXiv:2209.11055. ↗
  19. Bo Wang, Billy Dawton, Tsunenori Ishioka, and Tsunenori Mine. 2023. Optimizing answer representation using metric learning for efficient short answer scoring. The 20th Pacific Rim International Conference on Artificial Intelligence (PRICAI). find
  20. David M Williamson, Xiaoming Xi, and F Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice, 31(1):2–13. ↗
  21. Mengxue Zhang, Sami Baral, Neil Heffernan, and Andrew Lan. 2022. Automatic short math answer grading via in-context meta-learning. arXiv preprint arXiv:2205.15219. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fateen-2023-context-meta-learning-vs-semantic/