📄 CNLP-NITS-PP at GenAI Detection Task 2: Leveraging DistilBERT and XLM-RoBERTa for Multilingual AI-Generated Text Detection

Summary

This paper presents a team's system for Task 2 of the COLING 2025 GenAI Content Detection workshop. The task asks whether an essay was written by a person or by a large language model, and it has an English track (Subtask A) and an Arabic track (Subtask B). The human essays come from the ETS Corpus of Non-Native Written English. The machine essays were produced by seven open and closed LLMs. The authors tested several neural architectures and chose one per language: DistilBERT for English and XLM-RoBERTa for Arabic. Each transformer's embeddings are combined with handcrafted style features, such as vocabulary richness, stop-word frequency, sentence length and POS tags. The combined vector goes through a small dense head that outputs a probability. On the leaderboard, the English system reached 0.77 accuracy with 0.82 recall and placed 18th. The Arabic system reached 0.59 accuracy and placed 20th. The authors attribute the Arabic gap to using a general multilingual encoder rather than one adapted to Arabic. The work is a modest, reproducible baseline for detecting machine-written text in educational settings.

Key points

• The AI essays were generated by GPT-3.5-Turbo, GPT-4o, GPT-4o-mini, Gemini-1.5, Llama-3.1 (8B), Phi-3.5-mini and Claude-3.5. The human essays come from the ETS Corpus of Non-Native Written English (LDC2014T06).
• Data size and label balance:
  • English train has 2096 essays (629 human, 1467 AI); English dev has 1626 (1235 human, 391 AI).
  • Arabic train has 2070 essays (1145 human, 925 AI); Arabic dev has 481 (182 human, 299 AI).
• Before settling on DistilBERT and XLM-RoBERTa, the authors also tried CNN-LSTM, RNN, Bi-GRU, BERT, GPT-2 and RoBERTa.
• Training setup:
  • AdamW optimizer, learning rate 2e-5, batch size 16, 5 epochs.
  • Dropout 0.3, binary cross-entropy loss and a sigmoid output.
  • Checkpointing plus early stopping with patience 2.
• Test results:
  • DistilBERT on English: Acc 0.77, Precision 0.784, Recall 0.82, F1 0.77 (rank 18).
  • XLM-RoBERTa on Arabic: Acc 0.59, Precision 0.55, Recall 0.56, F1 0.55 (rank 20).

Interesting ideas

• I1. The system concatenates a transformer's sentence embedding with a few explicit, rule-computed style statistics before classification, so readable features sit alongside the opaque ones and can be inspected.
• I2. The distribution of POS tags is used as a signal of who or what produced a text, not just as an intermediate parsing result.
• I3. The English train and dev splits have nearly opposite class balance (mostly AI in train, mostly human in dev), which makes the separate precision and recall figures more informative than accuracy alone.
• I4. The authors' error analysis traces the weak Arabic score to an encoder that is multilingual but not specialised for the language, rather than to the classifier head.
• I5. A distilled, smaller transformer (DistilBERT) was chosen for speed and still produced the team's best English result.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Similar cheap statistics, such as stop-word rate, type/token ratio and POS profile, could flag LLM-teacher silver sentences whose style drifts away from the human treebank text, before those sentences enter training.
• I2 (tokens): The tagger's per-document tag histogram could serve as a lightweight, explainable genre/origin descriptor, letting later stages notice unusual input such as generated or translated text.
• I3 (learn): Evaluation suites could deliberately include splits with skewed label priors as a negative control, to show that a reader is not winning simply by guessing the majority answer.
• I4 (tree): gparse should prefer an English-specialised encoder, and when a parser underperforms, the encoder should be checked first before the parsing algorithm is tuned.
• I5 (tree): A distilled encoder could be tried as a faster backbone for gparse, which would make it cheaper to run several witness parsers when producing silver data.