📄 REGLAT at AraGenEval shared task: Morphology-Aware AraBERT for Detecting Arabic AI-Generated Text

Summary

The paper describes the REGLAT team's entry to the AraGenEval 2025 shared task. The task is a binary decision: was a piece of Arabic text written by a person or produced by a large language model? The organizers supplied a roughly balanced dataset with 4,798 training examples, 500 development examples and 500 test examples. Texts range from short social-media posts to articles of up to 512 tokens.

The system fine-tunes AraBERTv2 with a dropout-regularized classifier on the [CLS] vector. Before tokenization, a deterministic Arabic normalization step runs. It strips diacritics and tatweel, unifies Alef forms, maps Teh Marbuta and Alef Maksura to single forms, and cleans up whitespace. The authors report that this step roughly halves the number of distinct character combinations. Training is short: three epochs, with settings chosen by grid search.

On the test set the system reaches a weighted F1 of 0.63. Precision is higher than recall, which means the model leans towards labelling texts as human. Very short texts are the hardest. The authors argue that language-specific orthographic normalization matters for detecting machine-written text in morphologically rich languages.

Key points

• Data: 4,798 train / 500 dev / 500 test, about 50.3% human and 49.7% machine. A stratified 20% of the training data was held out for validation and early stopping.
• Model: AraBERTv2 (110M parameters, 12 layers, 768 hidden size, 12 heads, 64,000-token WordPiece vocabulary).
  • Max length 512, which leaves 99.3% of samples untruncated.
  • Dropout 0.3; with 0.5 the F1 fell by 2.1%, and with 0.1 the model overfit on long inputs.
• Normalization cut distinct character combinations from 847 to 423 per 1,000 tokens. It costs about 0.3 ms per sample.
• Training setup:
  • AdamW, learning rate 2×10⁻⁵, 10% warm-up, gradient clipping at 1.0, weight decay 0.01.
  • Batch size 8, which beat 16 and 32.
  • 3 epochs, about 3 hours on V100 GPUs, seed 42.
• Test results: F1 0.63, accuracy 0.65, precision 0.66, recall 0.60, specificity 0.69, balanced accuracy 0.65. Texts under 50 tokens were the weakest slice.

Interesting ideas

• I1. The paper quantifies normalization by how much it shrinks surface variety (distinct character combinations), not only by its effect on the end task; this gives a cheap, task-independent check of a preprocessing step.
• I2. Specificity is reported next to recall, which exposes that the classifier is conservative; the authors explain why wrongly flagging a human writer is the costlier mistake.
• I3. The maximum input length was chosen from a coverage statistic (the share of samples that fit untruncated) rather than by guessing.
• I4. The validation split preserves the class ratio, so early stopping and hyperparameter search are not misled by a skewed held-out set.
• I5. Errors are broken down by text length, which revealed that inputs under 50 tokens are much harder.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Mova's tokenizer could run a reversible normalization pass (quote styles, ligatures, spacing) that keeps the original span for explanations, and report the drop in surface types it achieves.
• I2 (learn): when silver trees or self-induced repair rules are accepted or rejected, Mova should report both the miss rate and the false-alarm rate, so a bias in either direction is visible.
• I3 (tree): gparse's sequence limit could be set from treebank sentence-length coverage, with the uncovered long tail routed to a separate fallback.
• I4 (learn): held-out sets for training the parsers and taggers could be stratified by genre or construction type, so model selection reflects the real mix.
• I5 (answer): QA and test-solving accuracy could be sliced by passage length, to find where short contexts leave the world model too thin to answer.