📄 BUSTED at ARATECT Shared Task: A Comparative Study of Transformer-Based Models for Arabic AI-Generated Text Detection

Summary

The BUSTED team from FAST Karachi took part in the AraGenEval shared task at ArabicNLP 2025 (the request calls it ARATECT). The task asks a system to decide whether an Arabic text was written by a person or generated by a machine. The team fine-tuned three pretrained encoders as binary classifiers on the 4,734-sample training set. Two are Arabic-specific (AraELECTRA and CAMeLBERT) and one is multilingual (XLM-RoBERTa). Only the AraELECTRA pipeline used heavy Arabic normalization, which unified letter variants and removed diacritics and non-alphanumeric symbols.

On the official test set, XLM-RoBERTa performed best with macro F1 0.7701, which placed the team 5th. CAMeLBERT scored 0.7290 and AraELECTRA 0.6180. The authors blame the weak AraELECTRA result mainly on normalization, which likely erased lexical and stylistic cues. They also observe that the two classes differ sharply in length and topic. The study is modest, but it makes two useful points: a language-specific model is not automatically the better choice, and preprocessing can remove exactly the signal a detector depends on.

Key points

• Data: the cleaned training set is nearly balanced, with 2,399 machine and 2,335 human samples.
  • Human texts average about 4,059 characters and machine texts about 1,935, so length alone is a strong but fragile cue.
  • Human texts focus on current geopolitics (Gaza, Israel, the war). Machine texts use more generic, formal phrasing, such as "international community" and "human rights."
• Models:
  • aubmindlab/araelectra-base-discriminator, with aggressive normalization
  • CAMeL-Lab/bert-base-arabic-camelbert-mix, with no normalization
  • xlm-roberta-base, with no normalization
• Shared setup: all three were trained with Hugging Face transformers, AdamW, learning rate 2e-5, batch size 4, weight decay 0.01, and maximum length 512. AraELECTRA and CAMeLBERT ran for 4 epochs and XLM-R for 5.
• Test results (F1 / recall / specificity):
  • XLM-RoBERTa: 0.7701 / 0.804 / 0.716
  • CAMeLBERT: 0.7290 / 0.780 / 0.640
  • AraELECTRA: 0.6180 / 0.728 / 0.372
  • AraELECTRA's very low specificity shows a strong lean toward the "machine" label.
• Inconsistent validation: only XLM-R had a stratified 80/20 split (3,787 / 947 samples). The two Arabic models were developed and evaluated on the full training set, so the comparison is not fully controlled.

Interesting ideas

• I1. Treat text normalization as an experimental variable, not a default cleanup, because collapsing character variants can wipe out the stylistic evidence a classifier needs.
• I2. Report specificity and recall alongside F1, since an aggregate score can hide a model that mostly predicts one class (as AraELECTRA did).
• I3. Profile the dataset before modelling to find shortcut features such as document length, and label them as brittle rather than as real competence.
• I4. Compare a broad multilingual encoder against narrow specialized ones under the same fine-tuning recipe, instead of assuming the specialized one wins.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): keep the raw surface form next to the normalized form and lemma in crate en, so later stages that need spelling or style cues can still reach them.
• I2 (learn): add per-class and confusion-style metrics to evaluation reports, so a parser or reader that over-predicts a frequent label is caught early.
• I3 (learn): run a trivial length-only or frequency-only baseline as a negative control, so any gain from the full pipeline can be shown to come from structure.
• I4 (tree): when choosing an encoder for gparse, benchmark a general multilingual backbone against English-only ones on our treebanks rather than picking by reputation.