📄 Osint at AraGenEval shared task: Fine-Tuned Modeling for Tracking Style Signatures and AI Generation in Arabic Texts

Summary

The authors describe their entry to AraGenEval 2025, a three-part Arabic shared task. The subtasks were rewriting formal MSA paragraphs in the voice of one of 21 writers, attributing a paragraph to one of those writers, and separating human-written Arabic from LLM output. Arabic makes all three harder because of diglossia, ambiguous rich morphology, and noisy orthography such as alef variants and inconsistent diacritics.

• Style transfer: they fine-tuned AraT5-base with an author tag and separator token prepended to the input. Preprocessing was kept minimal so stylistic quirks survive.
• Attribution: they put a small MLP head on AraBERTv2-base and added English round-trip backtranslation plus class-balanced batches.
• Detection: they joined mBERT's [CLS] vector with eight standardized surface statistics.

Reported scores are BLEU 19.87 / chrF 54.97, F1 0.79673 for attribution (on the validation split), and F1 0.75 for detection (8th place). The work offers a compact set of cheap baselines. However, its ablation table conflicts with the main results and cites a contrastive loss never described in the method, so those numbers deserve caution.

Key points

• Subtasks 1–2 use books by 21 writers (10 each), split into paragraphs and normalized to formal register with GPT-4o mini. ARATECT has 50,000/5,000/10,000 train/valid/test paragraphs, with machine text from GPT-4, Claude, Jais and others.
• Style transfer setup: lr 3e-5, batch 2, 3 epochs, 512-token limit. Plain 4-beam search (BLEU 19.87, chrF 54.97) narrowly outperformed a diversity-oriented decoding variant that divided 8 beams into 4 groups, used a 0.7 penalty and kept the shortest candidate (19.49 / 54.57).
• Attribution setup: head of 768→256, ReLU, dropout 0.3, then 21 classes, with 256-token inputs. Character n-gram subword embeddings gave no gain and were dropped. Final scores were F1 0.79673 and accuracy 0.83335.
• Detection setup: a 776-dim vector (768 + 8 features) feeds a 64-unit head. mBERT reached F1 0.75, precision 0.67, recall 0.86 and specificity 0.58. The AraBERTv2 variant collapsed (accuracy 0.498, specificity 0.156), labelling almost everything as machine-made.
• Error analysis:
  • Over-normalized, flattened style outputs.
  • Confusion between thematically similar authors (e.g., rural Egypt topics).
  • Formulaic human news flagged as AI.
  • Casual-toned LLM text accepted as human.

Interesting ideas

• I1. Controlling generation by prepending an explicit style/author token plus a separator lets one seq2seq model serve many target voices without separate models.
• I2. Concatenating a contextual embedding with a handful of standardized, human-readable surface features gives the classifier signals a person can inspect.
• I3. Reporting specificity and balanced accuracy, not just F1, exposed a detector that was effectively predicting one class.
• I4. Generating diverse candidates and picking by a naive criterion (shortest) did not help; diversity pays off only with a real scorer or reranker.
• I5. Class-balanced batch sampling counters skewed label distributions without new data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): the graph→text generator could take a register or genre tag token, so one realizer produces fable-style, school-test-style or plain output from the same meaning graph.
• I2 (learn): a silver-parse quality filter could fuse encoder states with interpretable features (sentence length, punctuation, witness disagreement), keeping its accept/reject decisions explainable.
• I3 (learn): evaluations with negative controls should always report specificity on the controls, so a reader or repair rule that says "yes" to everything is caught.
• I4 (back): candidate realizations should be reranked by round-trip parsing back to the graph and checking meaning preservation, not by length.
• I5 (tree): gparse training could balance batches across treebank sources or rare dependency relations, so small UD dialect treebanks and rare relations are not drowned out.