Elamur Research⌕
☰
home / papers / 2025 / CUET-NLP_Team_SS306 at AraGenEval Shared Task: A Transformer-based Framework for Detecting AI-Generated Arabic Text

CUET-NLP_Team_SS306 at AraGenEval Shared Task: A Transformer-based Framework for Detecting AI-Generated Arabic Text

Paper · 2025 · Sowrav Nath, Shadman Saleh, Kawsar Ahmed et al.

Summary

The paper addresses Task 3 of the AraGenEval 2025 shared task (ARATECT): deciding whether an Arabic text was written by a person or produced by an LLM such as GPT-4, Mistral or LLaMA. The CUET team fine-tunes several pretrained encoders: AraBERT, RoBERTa, Ara-RoBERTa, mBERT and MARBERT. Before training, the text is cleaned by stripping diacritics, unifying letter variants, and removing repeated characters and stray punctuation. Many documents are longer than the 512-token limit, so each one is cut into overlapping windows. Every window is classified on its own, and the window confidences are averaged per document to give the final label. On the official test set, AraBERT with this windowing reached F1 0.67 (up from 0.62), and the team placed 12th. On a held-out split of the training data the gains were larger. A breakdown by text length shows windowing helps most on long inputs, and Arabic-pretrained models behave more steadily than multilingual mBERT. The work is useful as a simple, reproducible baseline for machine-text detection in a morphologically rich, less-resourced language.

Key points

  • The ARATECT data has 4,798 balanced training samples (2,399 human, 2,399 machine) and 500 unlabeled test samples. Human texts come from Arabic news sites and verified literary sources.
  • Each text is split into chunks of 400 tokens with an overlap of 50 (max length 512), and document labels come from the mean chunk confidence. Training used batch 16, 5 epochs, learning rate 2e-5 and weight decay 0.001.
  • On the official test set, AraBERT rose from 0.62 to 0.67 F1 and RoBERTa from 0.58 to 0.61.
    • AraBERT's precision stayed near 0.5 while recall hit 0.97, so the model strongly leans toward the "machine" label.
  • On the custom split, AraBERT went from 0.79 to 0.87 and MARBERT from 0.80 to 0.87. mBERT fell from 0.81 to 0.60, with the drop concentrated on short texts (−0.58). Ara-RoBERTa gained +0.66 on long texts.
  • The chunked AraBERT got 431 of 500 texts right, but 69 human texts were still labeled as machine-generated.

Interesting ideas

  • I1. Splitting a long input into overlapping windows, scoring each separately, and pooling the scores lets a fixed-context encoder use the whole document instead of only its opening.
  • I2. Reporting scores separately for short, medium and long inputs exposed effects that the overall F1 hid, such as large gains concentrated on long texts.
  • I3. The same trick helped some models and badly hurt another (mBERT on short texts), so each tweak needs a per-component ablation rather than a blanket adoption.
  • I4. A detector with recall near 1 but precision near 0.5 is mostly defaulting to one class, which the confusion matrix makes visible.
  • I5. Normalizing orthographic variants (diacritics, letter forms, repeated characters) before modeling reduces sparsity from surface noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could process over-long sentences or multi-sentence spans in overlapping windows and reconcile predictions for tokens in the overlap zone, avoiding silent truncation.
  • I2 (learn): parser LAS and QA accuracy should be bucketed by sentence or story length so regressions on long inputs are not masked by easy short ones.
  • I3 (learn): each new repair rule or training change should be ablated per parser and per witness before it is admitted into the silver-data pipeline.
  • I4 (answer): readers' outputs on school tests could be checked for skew toward a default answer (e.g., always "yes" or the first option) using per-class confusion counts.
  • I5 (tokens): crate en could normalize Unicode quotes, dashes, ligatures and elongated spellings ("sooo") into a canonical form, keeping the original for explainability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chittagong University of Engineering & TechnologyChittagong · Bangladesh28 works · 29 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Sowrav Nath, Shadman Saleh, Kawsar Ahmed and Mohammed Moshiul Hoque. 2025. CUET-NLP_Team_SS306 at AraGenEval Shared Task: A Transformer-based Framework for Detecting AI-Generated Arabic Text. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.11

@inproceedings{nath2025cuet,
  title = {{CUET-NLP\_Team\_SS306 at AraGenEval Shared Task: A Transformer-based Framework for Detecting AI-Generated Arabic Text}},
  author = {Sowrav Nath, Shadman Saleh, Kawsar Ahmed and Mohammed Moshiul Hoque},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.11},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.11}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2019. Catboost: unbiased boosting with categorical features. Preprint, arXiv:1706.09516. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  3. Nuzhat Prova. 2024. Detecting ai generated text based on nlp and machine learning approaches. Preprint, arXiv:2404.10032. ↗
  4. Shadi Abudalfa, Saad Ezzini, Ahmed Abdelali, Hamza Alami, Abdessamad Benlahbib, Salmane Chafik, Mo El-Haj, Abdelkader El Mahdaouy, Mustafa Jarrar, Salima Lamsiyah, and Hamzah Luqman. 2025. The arageneval shared task on arabic authorship style transfer and ai-generated text detection. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics.
  5. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2025. Can ai-generated text be reliably detected? Preprint, arXiv:2303.11156. ↗
  6. Tokunaga Takenobu. 1994. Text categorization based on weighted inverse document frequency. Information Processing Society of Japan, SIGNL, 94(100):33–40. find
  7. Hamed Alshammari and Khaled Elleithy. 2024. Toward robust arabic ai-generated text detection: Tackling diacritics challenges. Information, 15(7). ↗
  8. Ye Zhang, Qian Leng, Mengran Zhu, Rui Ding, Yue Wu, Jintong Song, and Yulu Gong. 2024. Enhancing text authenticity: A novel hybrid approach for ai-generated text detection. Preprint, arXiv:2406.06558. ↗
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Araelectra: Pre-training text discriminators for arabic language understanding. Preprint, arXiv:2012.15516. ↗
  11. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794. ACM. ↗
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Preprint, arXiv:1911.02116. ↗
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  14. Md Zahid Hasan, Safiul Alam Sarker, MD Musa Kalimullah Ratul, Kawsar Ahmed, and Mohammed Moshiul Hoque. 2025. Cuet_nlp_finiteinfinity@ dravidianlangtech 2025: Exploring large language models for ai-generated product review classification in malayalam. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 599–604. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nath-2025-cuet-nlp-team-ss306-arageneval/