📄 CIOL at AraGenEval shared task: Authorship Identification and AI Generated Text Detection in Arabic using Pretrained Models

Summary

The authors, from CIOL, entered two subtasks of the AraGenEval shared task. Task 2 asks which of 21 Arabic authors wrote a paragraph. Task 3 asks whether an Arabic news text was written by a human or by an LLM. Both are treated as plain sequence classification with fine-tuned Arabic BERT encoders.

For authorship, CAMeL-Lab/bert-base-arabic-camelbert-mix gets a 21-way softmax head. Checkpoints are chosen by macro-F1 on validation. The data comes from ten books per author, split into paragraphs and paired with GPT-4o mini rewrites in a uniform formal register.

For detection, two AraBERT variants were compared, and the longer-context one with a custom head won. Authorship transferred well from validation to test, with test macro-F1 around 0.83. Detection collapsed, falling from about 0.98 F1 on validation to 0.66 on the official test.

The paper's main lesson is that a near-perfect score on in-distribution validation can be misleading. The authors blame domain shift and unseen generation styles, which makes cross-source evaluation essential for AI-text detectors.

Key points

• Task 2 setup: camelbert-mix, 4 epochs, AdamW (lr 2e-5), batch 16, weight decay 0.01. The paper is inconsistent about the input cap, giving 512 tokens in the system section and 256 in the setup section.
• Task 2 results:
  • Validation: accuracy 0.872, macro-F1 0.809.
  • Test: F1 0.827, accuracy 0.864.
• Task 3 data: 4,800 balanced training samples and 2,000 test samples. AI texts came from GPT-3.5, GPT-4 and Claude under varied prompts and generation parameters.
• Task 3 configurations:
  • Configuration 1: aubmindlab/bert-base-arabertv2, 512 tokens, dropout 0.3, gradient clipping, early stopping.
  • Configuration 2: bert-base-arabert with the built-in head, 256 tokens. It performed worse.
• Task 3 results:
  • Validation: F1 0.984, precision 1.0.
  • Test: F1 0.657, recall 0.568, specificity 0.840. The detector is cautious and misses many machine-written texts.

Interesting ideas

• I1. When validation scores look almost perfect, treat them as a warning, because a held-out set from the same sources can hide a large generalization gap.
• I2. Report specificity and balanced accuracy next to F1, since this combination exposes a classifier that rarely predicts the positive class.
• I3. Select checkpoints by macro-F1 instead of accuracy, so every class counts equally even when class sizes differ only mildly.
• I4. Compare a long input window against a short one as a controlled variant, and measure what context length is actually worth.
• I5. Build parallel pairs in which an LLM rewrites original paragraphs into one neutral formal register, separating what is said from how it is said.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Score each model on silver data drawn from text sources it never saw, next to the usual in-source split, so source overfitting shows up early.
• I2 (learn): Per-label precision versus recall for rare dependency relations or speech acts would reveal parsers or taggers that systematically under-predict them.
• I3 (tree): Selecting gparse checkpoints by macro-averaged per-deprel LAS would stop frequent relations from hiding regressions on rare ones.
• I4 (tree): A similar ablation of the gparse context window, sentence-only versus neighbouring sentences, would show whether extra context earns its memory cost.
• I5 (back): Such pairs could test whether the text → graph → text compressor preserves meaning while normalizing style.