📄 MarsadLab at TAQEEM 2025: Prompt-Aware Lexicon-Enhanced Transformer for Arabic Automated Essay Scoring

Summary

This paper describes MarsadLab's entry to TAQEEM 2025 Shared Task A. The task is to predict a holistic score (0–32) for student essays in Modern Standard Arabic, and evaluation uses prompts that do not appear in training (cross-prompt). The training data is small: 425 essays answering one explanatory and one persuasive prompt.

The authors extend an AraBERT-v2 regressor in two ways:

• A learned prompt-type vector is added to the pooled essay representation.
• Three numeric features come from a lexicon of 1–3-grams built from the training essays. Each lexicon entry is rated by how far the scores of essays containing it deviate from the average for its prompt type.

The combined 771-dimensional vector feeds a small regression head whose output is squeezed into the valid score range.

The system reached an average QWK of 0.438, clearly below the official baseline's 0.639. The authors blame simple concatenation and limited lexicon coverage for the gap. Still, they present the lexicon as an interpretable, genre-sensitive view of which vocabulary human raters tend to reward.

Key points

• Data: 425 training essays (215 explanatory, 210 persuasive), with human holistic scores from 2 to 31 on a 0–32 scale. Test prompts were 9 and 10; QWK served as the main evaluation measure, with RMSE reported alongside it as a supplementary measure.
• Lexicon statistics: each (n-gram, prompt type) pair records frequency, mean score, score standard deviation, number of distinct scores, and a z-score relative to the prompt-type mean. Importance is computed as log(1+count) × max(0, z), and only entries with positive importance are kept.
• Preprocessing:
  • Diacritics are removed and alif, ya and taa marbuta variants are unified.
  • Stopwords are kept on purpose.
  • Tokens appearing in the prompt text are removed to prevent leakage.
• Model:
  • AraBERT-v2 pooled vector (768) plus the prompt-type embedding, concatenated with 3 lexicon features: total importance, max importance, and importance-weighted mean z.
  • Head: linear layer → LayerNorm → dropout → linear → sigmoid scaled to [0, 32].
  • Training: MSE loss, AdamW at 2e-5, batch size 8, max length 512, early stopping on dev QWK.
• Results: QWK 0.447 / 0.428 on prompts 9 / 10, average 0.438 with RMSE 7.07. The baseline scored 0.639 QWK and 5.37 RMSE.

Interesting ideas

• I1. Words that are copied from the task prompt are dropped before computing word–score statistics, so the lexicon reflects the writer's own language rather than echoed instructions.
• I2. Function words and connectives are deliberately kept because their distribution shifts between explanatory and persuasive writing, making them cheap genre and reasoning signals.
• I3. Each lexical item is scored by a z-score against the mean of its own group, weighted by log frequency and clipped to positive values, which gives a ranked, human-readable list of quality-associated items.
• I4. Genre is signalled to the encoder by adding a learned type embedding to its representation, so one model can adapt to several text types.
• I5. The negative result suggests that three handcrafted scalars concatenated onto a 768-dimensional vector carry little weight; the authors point to attention-based or adaptive fusion as the fix.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when inducing word-to-label or word-to-correctness statistics from school-test data, first strip tokens shared with the question so silver statistics don't reward mere copying.
• I2 (frames): discourse markers such as concession and consequence connectives could be used as explicit cues for speech-act and argument-structure frames instead of being filtered out.
• I3 (learn): the same scoring could rank self-induced repair rules or candidate features per genre or treebank, keeping the ranking explainable and stopping very frequent items from dominating.
• I4 (tree): gparse could add a treebank or UD-dialect embedding so a single parser serves several source conventions while still exposing which convention it assumed.
• I5 (learn): when injecting explicit, explainable features into a transformer parser, use gated or attention-based fusion and test the contribution with ablations rather than trusting plain concatenation.