📄 BUSTED at AraGenEval Shared Task: A Comparative Study of Transformer-Based Models for Arabic AI-Generated Text Detection

Summary

This paper describes team BUSTED's entry to AraGenEval, a shared task on deciding whether an Arabic text was written by a person or produced by a machine. The authors fine-tuned three pretrained encoders as binary classifiers: AraELECTRA, CAMeLBERT and XLM-RoBERTa. They trained on the organisers' cleaned training set of 4,734 nearly balanced documents, using the Hugging Face transformers library and largely shared hyperparameters.

An initial look at the data found two differences between the classes. Human documents are about twice as long and focus on concrete current-affairs topics. Generated documents prefer generic, formal phrasing. Contrary to expectations, the multilingual XLM-RoBERTa beat both Arabic-specific models, reaching macro F1 0.7701 on the official test set and earning fifth place.

AraELECTRA was the only system run with heavy character normalization and diacritic removal, and it finished last. The authors blame that preprocessing for wiping out stylistic evidence. All systems had higher recall than precision, so they tended to mislabel human text as machine-made. The work is a reference point for detection in a lower-resource language and a warning about careless preprocessing.

Key points

• Data: the AraGenEval training set has 4,734 samples, with 2,399 machine and 2,335 human. Human texts average about 4,059 characters, while machine texts average about 1,935.
• Models: aubmindlab/araelectra-base-discriminator, CAMeL-Lab/bert-base-arabic-camelbert-mix and xlm-roberta-base.
• Official test results (macro F1 / accuracy):
  • XLM-RoBERTa: 0.7701 / 0.760
  • CAMeLBERT: 0.7290 / 0.710
  • AraELECTRA: 0.6180 / 0.550
  • AraELECTRA's specificity was especially low, at 0.372.
• Fine-tuning settings:
  • optimizer AdamW, learning rate 2e-5, weight decay 0.01
  • batch size 4, max sequence length 512
  • 4 epochs for the Arabic models, 5 for XLM-RoBERTa
• Setups were uneven across systems:
  • Only XLM-RoBERTa used a stratified 80/20 train/validation split (3,787 / 947).
  • The other two models were trained and checked on the full training set.
  • Only AraELECTRA received normalization: alef variants unified, ta marbuta mapped to ha, diacritics and non-alphanumerics removed.

Interesting ideas

• I1. Aggressive normalization can destroy the very cues a classifier needs, so stylistic signal should survive preprocessing rather than be "cleaned" away. — For Mova (tokens): keep the original surface form, casing and punctuation next to each lemma, so later stages can still see style and named-entity cues.
• I2. A quick profile of the classes before modelling exposed a strong shortcut: length alone separates the classes, as do topic n-grams. Shortcuts like these can inflate scores without any real understanding. — For Mova (learn): add length-only and n-gram-only baselines as negative controls, so a gain is credited to structure only when it beats these trivial predictors.
• I3. Reporting specificity and balanced accuracy alongside F1, and checking whether precision lags recall, shows in which direction a system is biased, not just how often it fails. — For Mova (answer): break test-reader errors down by direction (false "yes" vs. false "no"), so systematic over-acceptance is spotted and traced.
• I4. The comparison confounds several factors: model choice is tied to one preprocessing pipeline, and two models had no held-out evaluation. This shows why only one variable should change per comparison. — For Mova (tree): when the perceptron parser and gparse are compared, give them identical tokenization and the same stratified held-out treebank split, so LAS differences reflect the parsers alone.
• I5. A broadly pretrained multilingual encoder beat language-specific ones, suggesting that wide, diverse pretraining can generalize better than narrow specialization on a small task. — For Mova (learn): when picking an LLM teacher tier or witness parser for silver data, test broad models against domain-specialized ones empirically instead of assuming the specialized one wins.