📄 MISSION at AraGenEval Shared Task: Enhanced Arabic Authority Classification

Summary

The paper reports one participant's system for two subtasks of the AraGenEval shared task. Subtask 2 is Arabic authorship identification, a multi-class problem with a fixed set of candidate authors. Subtask 3 (ARATECT) is binary detection of human versus machine-written Arabic text.

For authorship, the author reused a pipeline that won an unrelated Kaggle math-problem classification contest. The backbone was replaced with ALLaM-7B-Instruct-preview, and a new prompt lists the candidate authors with numeric IDs. Training uses LoRA adapters, 4-bit quantization, mixed precision and a linear classification head. Because of compute limits, only one fold of stratified K-fold was trained.

For AI-text detection, AraModernBERT was fine-tuned with a fresh classification head. Its training data combined the official set with an external Arabic AI-generated corpus.

The systems placed 4th in Subtask 2 (Macro-F1 ≈ 0.84, accuracy ≈ 0.89) and 3rd in Subtask 3 (Macro-F1 0.80, accuracy 0.79). The work shows that recycling an existing pipeline and adding external data can be competitive on modest Kaggle hardware. All code is public.

Key points

• Data sizes:
  • Authorship (AID): 35,122 train, 4,157 dev and 8,413 test entries.
  • ARATECT: 17,604 train, 500 dev and 500 test entries, merged with the Al-Shaibani and Ahmed (2025) dataset.
• Subtask 2 setup:
  • Model: ALLaM-7B-Instruct-preview with LoRA, 4-bit quantization, distributed data parallelism and mixed precision.
  • Loss and validation: weighted cross-entropy for class imbalance, with micro F1 for validation.
  • Hardware: four NVIDIA L4 GPUs.
• Subtask 3 setup:
  • Model: AraModernBERT-Base-V1.0.
  • Training: max length 256 tokens, AdamW at learning rate 2e-5, 4 epochs, batch size 32 with dynamic padding.
  • Hardware: one Tesla P100.
• The official metric was Macro-F1, with accuracy as the secondary metric.
• Results:
  • Subtask 2: rank 4, F1 0.83753, accuracy 0.89053.
  • Subtask 3: rank 3, F1 0.80, accuracy 0.79.

Interesting ideas

• I1. The prompt lists every candidate label with a numeric ID, so the model only has to name a number, which turns open generation into a tightly constrained choice.
• I2. A whole training framework from a different winning competition was kept intact, and only the backbone and prompt were swapped, which saved engineering effort under tight deadlines.
• I3. External data of the same kind was merged with the official set to make the detector more robust than the in-domain data alone would allow.
• I4. Weighted cross-entropy compensates for uneven class frequencies, and Macro-F1 makes rare classes count equally in evaluation.
• I5. With limited compute, the author trained one stratified fold instead of all folds and still kept a class-balanced validation split.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): LLM teacher tiers could receive an enumerated inventory (e.g. VerbNet classes or UD relations) and answer with an index, making their silver labels trivially checkable and parseable.
• I2 (learn): when a new classifier is needed (speech acts, irony), first reuse an existing proven training harness and change only inputs and labels, so differences in results are attributable.
• I3 (tree): extra English treebanks, once converted to Mova's UD dialect, could be pooled with the core treebank to harden the perceptron parser and gparse.
• I4 (frames): rare semantic roles and speech-act types could be weighted in training and reported by macro-averaged F1, so common agent/patient roles do not hide failures.
• I5 (learn): quick experiments could use a single stratified fold over sentence types as a cheap but honest check before full cross-validation.