📄 GATech at AbjadGenEval Shared Task: Multilingual Embeddings for Arabic Machine-Generated Text Classification

Summary

The paper describes a system for the AbjadGenEval shared task, which asks whether an Arabic text was written by a person or produced by an AI model. Arabic detection tools lag behind English ones. The author fine-tuned the multilingual E5-large encoder with a classification head on the balanced competition set of 5,298 texts. Training used layer-wise learning rate decay, multi-sample dropout and Focal Loss. The main experiment compared ways of collapsing token vectors into one document vector: plain mean pooling, learned weighting of transformer layers, multi-head attention pooling, and gated fusion of several poolers. Mean pooling won with 0.75 F1 on the blind test set, and the learned aggregators scored lower. The author attributes this to the small data size: extra pooling parameters overfit, while a parameter-free average keeps the pretrained representations intact. The dataset also has a strong length gap between classes, which may let a model shortcut the task. The work is a useful reminder that simple choices can beat complex ones on small datasets.

Key points

• Backbone: multilingual-e5-large (24 layers, hidden size 1,024), max input of 512 tokens, effective batch size 64, learning rate 2e-5, LLRD factor 0.95, 2 epochs, cosine schedule with warmup.
• Data: 2,649 human and 2,649 machine texts. Human texts average 632 words and machine texts 303, so human texts are roughly twice as long.
• Test F1: mean pooling reached 0.75. Combinations of weighted layer pooling with attention, with or without gated fusion, landed around 0.70–0.71.
• The development set had only 2 samples. Every variant scored perfectly on it, so it could not separate the methods.
• Regularisation: five dropout masks (rates 0.1 to 0.3) are applied in training and their logits averaged. Focal Loss shifts weight toward hard examples.

Interesting ideas

• I1. When labelled data is scarce, a parameter-free aggregation such as averaging can generalise better than learned attention or gating, because it adds nothing that can overfit.
• I2. Averaging logits over several dropout masks of different rates gives a small in-model ensemble at no extra inference cost.
• I3. A large length gap between classes can become a hidden shortcut, so the data should be inspected for trivial signals before trusting the model's score.
• I4. A tiny development set that every variant aces gives no basis for choosing between them, and the real differences only appeared on the test set.
• I5. Smaller learning rates for lower encoder layers help keep pretrained knowledge while the upper layers adapt to the task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): In gparse, merging subword vectors into word vectors could start from plain averaging, with learned poolers adopted only if they beat it on held-out treebank LAS.
• I2 (tree): This could be tried as a cheap regulariser when training gparse on small UD treebanks or converted dialect data.
• I3 (learn): Evaluations could include a negative-control baseline that sees only surface statistics such as length, to show the model's gains come from structure.
• I4 (learn): Model-selection sets for parser or reader variants should be large enough to separate candidates, and runs where all variants score perfectly should be flagged.
• I5 (tree): LLRD could be applied when fine-tuning the transformer under gparse, protecting general representations while the top layers learn dependency arcs and labels.