Elamur Research⌕
☰
home / papers / 2025 / IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry

IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry

Paper · 2025 · Mohammad AL-Smadi

Summary

The paper describes a system for the COLING 2025 GenAIDetect shared task (Task 2), which asks whether an academic essay was written by a person or a generative model. The task has an English track and an Arabic track. The human-written essays are drawn from ETS's collection of English essays by non-native writers. Machine essays were produced by seven LLMs, including GPT-4o, Gemini-1.5, Llama-3.1, Phi-3.5-mini and Claude-3.5.

The author fine-tunes ELECTRA discriminators for English and AraELECTRA for Arabic. Each model is paired with seven simple stylometric measurements, such as word and sentence counts, type-token ratio, average word length and punctuation counts. A batch-normalised feature branch and the [CLS] embedding branch are concatenated before a final classifier.

Results on the test set:

  • English: ELECTRA-Large reached 99.7% F1, which ranked 2nd of 26 teams.
  • Arabic: AraELECTRA reached 98.4% F1, which ranked 1st of 23 teams.

The organisers' TF-IDF + SVM baseline stayed below 48%. Removing the stylometric features cost a few F1 points. The work shows that a compact encoder plus a handful of cheap, interpretable signals can separate LLM-written essays from human ones in two typologically different languages.

Key points

  • Data sizes: The Arabic training set has 2,070 essays (925 AI / 1,145 human). The English training set has 2,096 essays and is skewed toward AI (1,467 AI / 629 human). The test sets contain 293 Arabic and 1,130 English essays.

  • Test F1 by model:

    ModelLanguageTest F1
    AraELECTRA-baseArabic98.4%
    ELECTRA-smallEnglish98.5%
    ELECTRA-largeEnglish99.7%
    Baseline (TF-IDF unigrams, max 10,000 features, SVM)Arabic46.1%
    Baseline (TF-IDF unigrams, max 10,000 features, SVM)English47.8%

  • Ablation: Without the stylometric features, test F1 fell to 96.9% for Arabic and 96.1% for English ELECTRA-small. This is a loss of roughly 1.5 and 2.4 points.

  • Architecture: The [CLS] vector and the BatchNorm1d-normalised numeric features each pass through their own linear layer with ReLU. The two outputs are concatenated and fed to a final linear layer, with dropout for regularisation.

  • Training: Models were trained for up to 10 epochs with early stopping at patience 2. Training and validation loss stayed close throughout. On the validation set, English was classified perfectly and Arabic had a single error.

Interesting ideas

  • I1. A two-branch design that joins a contextual [CLS] embedding with a small, normalised vector of hand-crafted counts lets transparent surface statistics add measurable signal on top of a black-box encoder.
  • I2. The paper measures the value of the extra features by retraining the same model without them and reporting the F1 gap, which turns a design claim into a quantified one.
  • I3. Keeping the organisers' bag-of-n-grams baseline in the results table exposes how much of the task cannot be solved by word statistics alone.
  • I4. Comparing ELECTRA-Small with ELECTRA-Large shows the small model is nearly as accurate at a fraction of the compute, so model size becomes an explicit trade-off rather than a default.
  • I5. Cheap stylometric cues such as type-token ratio, average sentence length and comma counts can help tell machine-written text from human writing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could receive a few explicit, named features (e.g., sentence length, punctuation density) through a separate normalised branch, so their contribution to parse decisions can be inspected and reported.
  • I2 (learn): every added signal in Mova's training pipeline could be shipped with a matching "without this signal" run, acting as a negative control that shows the gain is real.
  • I3 (learn): reporting a deliberately naive bag-of-words floor next to Mova's structured readers would make it visible where the tree-and-frames route actually pays off.
  • I4 (tree): gparse could be benchmarked in small and large variants on the same treebanks to choose the cheapest model whose LAS stays within an acceptable margin.
  • I5 (learn): such a lightweight style check could flag silver sentences or LLM teacher outputs whose style drifts far from human treebank text, before they enter training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 works

Cite

Cite as Mohammad AL-Smadi. 2025. IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry. Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alsmadi2025integrityai,
  title = {{IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry}},
  author = {Mohammad AL-Smadi},
  booktitle = {Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect)},
  year = {2025},
  url = {https://aclanthology.org/2025.genaidetect-1.31/}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Kutbi, Ali H. Al-Hoorie, and Abbas H. Al-Shammari. 2024. Detecting contract cheating through linguistic fingerprint. Humanities and Social Sciences Communications, 11:1–9. ↗
  2. Mohammad Al-Smadi. 2023. Chatgpt and beyond: The generative ai revolution in education. arXiv preprint arXiv:2311.15198. ↗
  3. Hamed Alshammari, Ahmed El-Sayed, and Khaled Elleithy. 2024. Ai-generated text detector for arabic language using encoder-based transformer architecture. Big Data and Cognitive Computing, 8(3):32. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Araelectra: Pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516. ↗
  5. Wissam Antoun, Virginie Mouilleron, Benoît Sagot, and Djamé Seddah. 2023a. Towards a robust detection of language model generated text: Is chatgpt that easy to detect? arXiv preprint arXiv:2306.05871. ↗
  6. Wissam Antoun, Benoît Sagot, and Djamé Seddah. 2023b. Data-efficient french language modeling with camemberta. arXiv preprint arXiv:2306.01497. ↗
  7. Shammur Absar Chowdhury, Hind Al-Merekhi, Mucahid Kutlu, Kaan Efe Keleş, Fatema Ahmad, Tasnim Mohiuddin, Georgios Mikros, and Firoj Alam. 2025. GenAI content detection task 2: AI vs. human – academic essay authenticity challenge. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. ↗
  8. K Clark. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  9. A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  10. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315– 323. JMLR Workshop and Conference Proceedings. find
  11. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Y. Zou. 2023. Gpt detectors are biased against non-native english writers. Patterns, 4. ↗
  12. Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364. ↗
  13. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. ↗
  14. Chidimma Opara. 2024. Styloai: Distinguishing ai-generated content with stylometric analysis. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, pages 105–114, Cham. Springer Nature Switzerland. find
  15. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  16. Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltỳnek, Jean Guerrero-Dib, Olumide Popoola, Petr Šigut, and Lorna Waddington. 2023. Testing of detection tools for ai-generated text. International Journal for Educational Integrity, 19(1):26. ↗
  17. Hin Boo Wee and James D Reimer. 2023. Non-english academics face inequality via ai-generated essays and countermeasure tools. BioScience, 73(7):476–478. ↗
  18. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. ↗
  19. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. ↗
  20. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smadi-2025-integrityai-genai-detection-task-2/