📄 912 at TAQEEM 2025: A Distribution-aware Approach to Arabic Essay Scoring

Summary

This system paper addresses holistic Arabic automatic essay scoring (TAQEEM 2025, Task A), where performance is judged by Quadratic Weighted Kappa (QWK). The training data is small (426 essays across an explanatory and a persuasive prompt), and its scores pile up in the middle range. A plain regression loss therefore tends to pull predictions toward the average. The authors fine-tune the AraBERT v0.2 encoder and study three things one at a time: the depth of the regression head (linear vs. small MLPs), the training objective, and whether the prompt text is fed in alongside the essay. Their main change is a Weighted MSE that scales each example's squared error by the inverse frequency of its gold score. Rare high and low scores therefore count for more. They compare it with plain MSE and with MSE plus a differentiable "soft" QWK term. The best setup (Weighted MSE, one hidden layer, essay text only) beats the official baseline on both the development folds and the private test set. The broader lesson is that matching the loss to the label distribution can matter more than adding architectural complexity.

Key points

• Data: 426 training essays and 840 test essays, two prompt types, with a bell-shaped score distribution concentrated around roughly 18–25. The backbone is AraBERT v0.2, trained with AdamW (lr 5e-5, batch 16, up to 100 epochs) and selected on average dev QWK.
• Weighted MSE uses a weight of 1/N_s for each integer score s. The weights are computed once over the training set and looked up by gold label.
• Dev results (average QWK over two folds):
  • Baseline linear head with MSE: 0.716.
  • Best model (wMSE, k=1): 0.766.
  • Best plain MSE configuration (k=3): 0.761.
  • Best MSE+QWK configuration (k=2): 0.747.
  • Gaps among the stronger configurations are fairly small.
• Private test set: official QWK 0.673 against the baseline's 0.639, and RMSE 5.333 against 5.372.
• Ablations on the input and on head depth:
  • Adding the prompt before the essay lowered dev QWK to 0.759.
  • With wMSE, deeper heads (k=2, 3) did slightly worse than k=1.
  • Errors are mostly confusions between neighbouring score bins. Well-written but off-topic essays tend to receive overly generous scores.

Interesting ideas

• I1. Weighting each training example by the inverse frequency of its gold label is a cheap fix for a model that collapses toward the majority, and it requires no resampling or architecture change.
• I2. A non-differentiable agreement metric can be made trainable by converting each continuous prediction into a softmax over nearby classes and building a "soft" confusion matrix from those vectors.
• I3. The best head size depended on the loss: reweighting appeared to need slightly more head capacity than a linear layer, while deeper heads overfit the small dataset.
• I4. The ablations start from the best configuration and change a single axis at a time (loss, head depth, input), which keeps each comparison interpretable.
• I5. The error analysis showed that a model trained on general writing quality rewards fluent text that ignores the task, a failure that a holistic score hides.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): The same per-label weighting could be applied to rare UD relations (e.g. reparandum, dislocated, vocative) when training the perceptron parser or gparse, so they are not drowned out by nsubj/obj.
• I2 (learn): A soft confusion matrix over predicted vs. gold labels could provide a differentiable training signal when tuning scorers in the LLM-teacher pipeline, rather than relying only on a final discrete metric.
• I3 (learn): When the loss or label weighting is changed for gparse, the classifier-head depth should be re-swept along with it rather than fixed in advance, since the optimum may shift.
• I4 (learn): Evaluation reports for new silver-data or repair-rule variants could use this one-change-from-best layout, paired with negative controls, so the source of each gain is clear.
• I5 (answer): The school-course readers should check explicitly whether an answer's frames actually address the question's frame, instead of trusting fluency or surface overlap.