📄 Phantoms at BAREC Shared Task 2025: Enhancing Arabic Readability Prediction with Hybrid BERT and Linguistic Features

Summary

This system paper targets sentence-level readability prediction for Modern Standard Arabic in the BAREC 2025 Shared Task. The authors entered the strict track, so they trained only on the BAREC Corpus training split. Each sentence must be placed into one of 19 ordinal difficulty levels. Their model joins two signals. One is the [CLS] embedding from AraBERTv02. The other is a standardized vector of hand-built linguistic features, which are concatenated with the embedding and fed to a small MLP classifier. Starting from more than 200 features computed with camel-tools, they reduced the set to 100 in three steps: a variance cutoff, removal of highly correlated pairs, and Random Forest importance ranking. The features cover surface statistics, characters, morphology, semantics and lexical properties. Evaluated on the blind test set, the system reached 82.7% QWK and 57.6% exact-match accuracy, and most of its errors missed by a single level. The work suggests that explicit linguistic cues can usefully support a pretrained encoder for a fine-grained ordinal task in a morphologically rich language. However, the main text includes no feature ablation that would isolate their contribution.

Key points

• Encoder: aubmindlab/bert-base-arabertv02. Its 768-dim [CLS] vector is concatenated with 100 scaled features, giving a 768+100 dimensional input. The head is Linear → SiLU → dropout 0.2 → Linear, with dropout 0.3 applied to the fused vector.
• Preprocessing with camel-tools:
  • Normalization covers Unicode, Alef, Alef Maksura and Teh Marbuta, and all diacritics are stripped.
  • Morphological features are taken from the top analysis of an MLE disambiguator.
  • Feature groups include POS/gender/number/aspect/case proportions, affix ratios, verb-to-noun ratio, NER counts, sentiment, MSA dialect-ID confidence, stop-word ratio and stem diversity.
• Feature selection runs in three stages:
  1. Drop features with variance below 0.01.
  2. Drop one feature from any pair with Pearson correlation above 0.95.
  3. Keep the top 100 by Random Forest importance.
• Training setup:
  • AdamW with learning rate 5e-5, a linear schedule and batch size 16.
  • At most 10 epochs, with mixed precision.
  • Cross-entropy weighted by inverse class frequency.
  • Early stopping on validation QWK with patience 2.
• Blind test results: QWK 82.7%, exact accuracy 57.6%, Acc ±1 72.3%, MAE 1.06. Accuracies after collapsing to 3, 5 and 7 coarser levels range from 67.4% to 77.2%.

Interesting ideas

• I1. Late fusion of an encoder's pooled vector with a scaled vector of interpretable hand-made features lets a neural model use explicit linguistic cues without redesigning the network.
• I2. A cheap three-step pruning pipeline (variance cutoff, correlation de-duplication, tree-based importance ranking) turns a sprawling feature inventory into a compact, non-redundant set.
• I3. Ordinal labels are scored with metrics that respect distance between levels: QWK, off-by-one accuracy, MAE, and accuracy after merging levels into 3, 5 or 7 bands. This separates near misses from gross errors.
• I4. Interpretable difficulty signals such as stem diversity, long-word ratio, affix ratios and verb-to-noun ratio can be computed directly from morphological analysis, with no learned model needed.
• I5. Weighting the loss by inverse class frequency keeps rare classes from being ignored when labels are badly skewed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could take explicit morphological or lexicon features from crate en alongside its transformer states, so that some of its decisions can be traced to named, explainable inputs.
• I2 (learn): the same pipeline could prune templates in the perceptron parser or the features behind self-induced repair rules, removing near-duplicates before training.
• I3 (answer): when Mova grades or places school-course texts and test items by difficulty, these metrics would show whether its mistakes are small slips or serious misjudgments.
• I4 (tokens): crate en could expose these statistics from its lemmas and morphology, giving the readers an explainable estimate of how hard a sentence is.
• I5 (learn): weighting rare dependency relations or rare frame roles this way when training on silver data could stop frequent labels from dominating.