📄 ARxHYOKA at TAQEEM2025: Comparative Approaches to Arabic Essay Trait Scoring

Summary

This paper describes the ARxHYOKA entry to TAQEEM2025 Task B. The task is to score Arabic student essays on seven traits from the CAST rubric: Relevance is rated from 0 to 2, while the remaining six (Vocabulary, Style, Organization, Grammar, Development and Mechanics) are each rated from 0 to 5. The hard part is that test essays answer writing prompts that never appear in training.

The team compared four system families:

• GPT-4.1 scoring with a structured chain-of-thought prompt and in-context examples;
• encoder fine-tuning (mDeBERTa-v3-base, XLM-R-large, CAMeLBERT-mix) with a joint seven-output regression head;
• classical regressors on CAMeLBERT-mix embeddings, optionally with 14 handcrafted linguistic features;
• LoRA adaptation of the Arabic generator Saka-14B on hand-written rubric reasoning chains.

All tuning used a cross-genre protocol (explanatory essays for training, persuasive for evaluation), followed by retraining on both genres. On the official test, GPT-4.1 with 10 examples gave the best mean QWK (0.612 against a 0.472 baseline) and also among the smallest errors. Every family beat the baseline on the test data. The work offers an inexpensive, side-by-side picture of which modelling choices carry over to unseen prompts in Arabic AES.

Key points

• Data: development phase used prompts 1 (explanatory, 215 essays) and 2 (persuasive, 210). Testing used prompts 9 and 10 (420 essays each). Essays are about 300 words and were rated by experts against the CAST rubrics.
• GPT-4.1 prompting:
  • Switching rubric and prompts to English (translated by GPT-4.1) raised QWK from 0.539 to 0.579.
  • During experiments, 0/1/5/10 shots gave 0.579/0.597/0.603/0.631.
  • All API use cost roughly USD 21 for about 21.3M tokens.
• Development phase (mean QWK):
  • baseline 0.582
  • mDeBERTa-v3-base (lr 2e−5, last 6 layers trainable) 0.575
  • GPT 10-shot 0.564
  • Ridge with AVG pooling plus linguistic features 0.539
  • Saka-14B with LoRA 0.480
• Official test (mean QWK):
  • GPT 10-shot 0.612, GPT 1-shot 0.610 (lowest MSE 0.758 and RMSE 0.845)
  • classical ML 0.582
  • fine-tuned BERT 0.554
  • Saka-14B 0.538
  • baseline 0.472
• Error tendencies:
  • GPT had the most exact matches.
  • BERT and Saka-14B tended to score too high, especially on Vocabulary and Style.
  • Ridge tended to score too low, most clearly on Organization and Development.

Interesting ideas

• I1. Choose every hyperparameter and prompt under the same kind of shift expected at test time (one genre in, another out), and only then retrain on everything; this makes model selection honest about transfer.
• I2. Alongside the agreement metric, report per category the share of exact, too-high and too-low predictions, which exposes systematic bias that a single score hides.
• I3. Appending a small set of shallow, interpretable linguistic counts to dense embeddings gave a consistent lift for simple linear regressors.
• I4. The language in which the rubric and instructions are written measurably changed the LLM's scoring quality, so instruction language is a tunable variable worth testing on held-out data.
• I5. Force a strict output format, then parse it mechanically, and snap continuous predictions to the valid range of each field by rounding and clamping.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when picking repair rules or parser settings, train on one text genre (say, tales) and score on another (say, school tests) before merging the silver data, so gains that only hold in-genre are filtered out.
• I2 (learn): evaluation reports for witness parsers and students could split errors by direction per label (e.g. attaching too high vs. too low, overcounting vs. undercounting quantities), turning a flat accuracy into actionable repair targets.
• I3 (tokens): the tagger and morphology layer could expose explicit, explainable features (affix, length, frequency cues) next to learned representations, keeping decisions traceable while gaining accuracy.
• I4 (learn): for LLM teacher tiers, systematically compare instruction wordings and formats on a small gold set before trusting a teacher's silver labels.
• I5 (answer): readers answering tests or word problems could emit structured, machine-checkable answers and project them onto the set of admissible values (e.g. integer counts, allowed options), catching malformed outputs early.