📄 Sebaweh at AraGenEval Shared Task: BERENSE - BERt based ENSEmbler for Arabic Authorship Identification

Summary

The paper describes the winning system for Subtask 2 of the AraGenEval 2025 shared task. The task is to work out which of 21 Arabic writers produced a given paragraph. The data is hard: it mixes Classical Arabic, MSA and Egyptian dialect, covers many genres and periods, and is heavily imbalanced, with some authors having under 100 samples and others over 3,000.

The team first tried several designs, including RNN heads, frozen embeddings with SVM or Random Forest, and topic-feature concatenation. Plain fine-tuning with a fully connected softmax head worked best. They then:

• fine-tuned four Arabic-capable encoders with this head and combined them by averaging their probabilities (soft voting);
• cleaned the data by removing duplicates, splitting very long texts and stripping diacritics;
• added extra books from the Hindawi Books dataset for poorly represented authors;
• used an inverse-frequency class-weighted loss.

The ensemble reached a test macro-F1 of 0.9046 and took first place, about 15 points above their RNN baseline. The work shows that a carefully assembled ensemble of standard fine-tuned encoders can handle multi-genre, multi-variety Arabic authorship attribution without hand-crafted stylometric features.

Key points

• Ensemble members: AraBERT v0.2 (136M), CAMeLBERT-Mix (110M), Arabic XLM-RoBERTa (270M) and GATE-AraBERT (135M), combined by soft voting. Arabic-labse-Matryoshka and Arabic distilbert-base were dropped because their validation F1 was below 0.80.
• Validation macro-F1 by architecture:
  • BERT+RNN baseline: 0.75
  • frozen BERT + bagging SVM: 0.66
  • frozen BERT + Random Forest: 0.35
  • BERT + fully connected layer: 0.85
  • ensemble: 0.90
• Official test results: macro-F1 0.9046, accuracy 0.9327, precision 0.9012, recall 0.9143.
• Preprocessing:
  • 2,740 duplicates removed;
  • 1,381 texts longer than 3,000 characters split into chunks of roughly 2,000 characters;
  • diacritics stripped;
  • the validation set merged into training before the final test run.
• Effect of each addition:
  • Hindawi Books augmentation raised validation F1 for most models: CAMeLBERT 0.84→0.90, GATE 0.84→0.89, AraBERT 0.88→0.90, with no change for XLM-RoBERTa.
  • Class weighting mostly had little effect. It helped frozen GATE + SVM (0.56→0.66) and slightly helped XLM-RoBERTa (0.82→0.83).
  • Adding topic features left CAMeLBERT unchanged at 0.85.

Interesting ideas

• I1. Long texts were cut into chunks near the mean length so the classifier could not rely on document length as a shortcut, a useful guard against spurious surface cues.
• I2. Probability distributions from several differently pretrained encoders were averaged, so each model's distinct errors partly cancel out.
• I3. Candidate models had to clear a fixed validation threshold before joining the ensemble, which keeps weak members from diluting the vote.
• I4. Extra features (topic keywords) were tested in an ablation and dropped when they added nothing beyond contextual embeddings, which keeps the design lean and honest.
• I5. The authors admit their features may reflect content rather than style, and confusions clustered among authors with similar genres or eras. This is a reminder to check what signal a model actually uses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when evaluating, we could add negative-control splits where sentence length is equalized, to check that our parsers and readers do not exploit length rather than structure.
• I2 (tree): we could average per-token head and label distributions from the perceptron parser and gparse, and record each model's contribution so the merged arc stays explainable.
• I3 (learn): we could require each witness parser to pass a minimum LAS on a held-out treebank before its vote counts toward silver data agreement.
• I4 (frames): before adding any new cue to role assignment (e.g., an extra VerbNet feature), we could run a with/without comparison and keep the cue only if it changes outcomes measurably.
• I5 (answer): for fable and tale QA, we could analyze confusions by topic to detect when a reader answers from topical overlap instead of from the event structure of the text.