📄 NLP_wizard at AraGenEval shared task: Embedding-Based Classification for AI Detection and Authorship Attribution

Summary

This system paper reports one team's entry to the AraGenEval shared task at ArabicNLP 2025. The team took part in two Modern Standard Arabic subtasks: identifying which of 21 authors wrote a passage, and deciding whether a passage was written by a human or generated by an LLM (the ARATECT dataset). The design deliberately avoids fine-tuning. The authors run a frozen xlm-roberta-large encoder, build one document vector per text, and train default scikit-learn linear classifiers on those vectors: RidgeClassifierCV for AI detection and LinearSVC for authorship. They compare two ways of forming the vector. One averages every token embedding, the other takes the final-layer [CLS] vector. Averaging won by roughly three F1 points on both tasks, reaching macro-F1 0.7400 for detection and 0.8130 for attribution. Each result placed 10th on its leaderboard. The work matters as a cheap and reproducible baseline that runs on a single consumer GPU. It also gives evidence that style signals are spread across tokens and are not concentrated in a summary position.

Key points

• Authorship data covers 21 authors with 10 public books each, split into paragraphs. ARATECT mixes curated human news and literary texts with outputs from Mistral, GPT-4 and LLaMA.
• Inputs were truncated to 512 tokens. Embeddings were computed once and cached, and every model ran with default hyperparameters on one RTX 4060 (8GB).
• Blind-test macro-F1 for AI detection was 0.7400 with mean pooling and 0.7100 with [CLS].
• Blind-test macro-F1 for authorship was 0.8130 with mean pooling and 0.7830 with [CLS]. Both mean-pooling runs ranked 10th.
• No extra annotated data, lexicons, handcrafted features or rule-based preprocessing were used, only the official train/dev/test splits.

Interesting ideas

• I1. Encode the corpus once with a frozen encoder, cache the vectors, and then sweep many cheap classifiers over the cache, which makes iteration nearly free.
• I2. Run a controlled comparison of pooling choices where only the pooling changes, which isolates where the signal actually lives in the representation.
• I3. Treat stylistic properties as diffuse, spread over function words, rhythm and punctuation, and aggregate over all tokens instead of trusting one summary slot.
• I4. A frozen embedding fed into a linear model makes a strong, cheap reference point that any more elaborate system should beat.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Cache gparse encoder states for a treebank once, then train and compare light probes (e.g. for tag or relation hints) without re-running the transformer each time.
• I2 (tree): When gparse builds span or sentence vectors, benchmark mean versus single-position pooling under identical settings before committing to an architecture.
• I3 (frames): Register and irony cues for prag could be scored by aggregating per-token evidence across the whole utterance rather than reading a single head word.
• I4 (learn): Add such a bag-free embedding probe as a baseline control in evaluations, so we can show how much the explainable tree-to-frames pipeline adds on tasks like school tests.