Elamur Research⌕
☰
home / papers / 2025 / CIC-NLP at GenAI Detection Task 1: Leveraging DistilBERT for Detecting Machine-Generated Text in English

CIC-NLP at GenAI Detection Task 1: Leveraging DistilBERT for Detecting Machine-Generated Text in English

Paper · 2025 · Tolulope Olalekan Abiola, Tewodros Achamaleh Bizuneh, Oluwatobi Joseph Abiola et al.

Summary

The CIC-NLP team took part in Subtask A of Task 1 at the GenAI Content Detection workshop (COLING 2025). The task is a binary decision: was an English text written by a person or produced by an LLM? The organizers supplied a large corpus drawn from many human and machine sources. It has 610,767 training texts (about 62% machine-made) and 261,758 development texts with a similar class skew. The team fine-tuned distilbert-base-uncased with a classification head added on top and did not engineer any features. Training used several standard Hugging Face settings:

  • cosine learning-rate decay with warm-up
  • mixed precision
  • gradient accumulation
  • dynamic padding
  • early stopping and checkpointing keyed to F1

On the development set the model reached about 0.95 accuracy and F1. On the hidden test set it scored a micro-F1 of 0.7068. The authors present this as evidence that a compact transformer can handle machine-text detection reasonably well. The large drop from development to test also shows how fragile in-distribution validation is for this kind of task.

Key points

  • Data: the training set has 381,845 machine texts and 228,822 human texts. The development set has 163,430 machine texts and 98,328 human texts. Label 1 means human, and label 0 means AI.
  • Model and training: distilbert-base-uncased with a binary classification layer, trained for 3 epochs. Gradients were accumulated over 2 batches, with mixed precision, a cosine schedule with warm-up, and DataCollatorWithPadding.
  • Development curve: F1 rose from 0.9263 to 0.9434 to 0.9461 over the three epochs. Validation loss went up at epoch 3 (0.1509 → 0.1807) even though training loss kept falling.
  • Test results on Codabench: micro-F1 0.7068 and macro-F1 0.7242.
  • Related work cited:
    • Li et al.: detection gets much harder on out-of-distribution data, and mixing many domains and LLMs blurs the linguistic differences between human and machine text.
    • Fernández-Hernández et al.: adding readability, sentiment and toxicity metadata to a fine-tuned BERT lowered performance.

Interesting ideas

  • I1. The best checkpoint was chosen by the task metric (F1) rather than by loss, which matters because validation loss rose at epoch 3 while F1 still improved.
  • I2. Dynamic per-batch padding pads each batch only to its longest sequence, so less compute is spent on filler tokens.
  • I3. A development score near 0.95 fell to about 0.71 on the test set, which warns that a validation split drawn from the same distribution can badly overstate real performance.
  • I4. The cited finding that mixing domains and generators shrinks the POS and named-entity differences between human and machine text suggests that machine text is not cleanly separable by shallow features.
  • I5. The cited result that extra metadata features hurt a fine-tuned BERT shows that adding signals does not automatically help a strong model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's training loops should keep the checkpoint with the best LAS or best answer accuracy on held-out data, because loss alone could discard a better model.
  • I2 (tree): sentence lengths in treebanks vary widely, so gparse training could batch by length and pad dynamically to speed up GPU passes.
  • I3 (learn): evaluation should include held-out genres or sources as a control, so that silver-data gains are not judged only on data that resembles the training data.
  • I4 (learn): when LLM teacher tiers produce silver data, check per source whether teacher-written examples drift in style from human treebank text before mixing them.
  • I5 (frames): each new pragmatic cue (sentiment, irony markers) should be ablated one at a time with an explicit before-and-after score rather than added all at once.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Tolulope Olalekan Abiola, Tewodros Achamaleh Bizuneh, Oluwatobi Joseph Abiola, Temitope Olasunkanmi Oladepo, Olumide Ebenezer Ojo, Grigori Sidorov and Olga Kolesnikova. 2025. CIC-NLP at GenAI Detection Task 1: Leveraging DistilBERT for Detecting Machine-Generated Text in English. Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abiola2025cic,
  title = {{CIC-NLP at GenAI Detection Task 1: Leveraging DistilBERT for Detecting Machine-Generated Text in English}},
  author = {Tolulope Olalekan Abiola, Tewodros Achamaleh Bizuneh, Oluwatobi Joseph Abiola, Temitope Olasunkanmi Oladepo, Olumide Ebenezer Ojo, Grigori Sidorov and Olga Kolesnikova},
  booktitle = {Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect)},
  year = {2025},
  url = {https://aclanthology.org/2025.genaidetect-1.29/}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alberto Fernández-Hernández, Juan Luis Arboledas-Márquez, Julián Ariza-Merino, and Salud María Jiménez Zafra. 2023. Taming the turing test: Exploring machine learning approaches to discriminate human vs. ai-generated texts. In IberLEF@ SEPLN. find
  2. Olaronke Oluwayemisi Adebanji, Olumide Ebenezer Ojo, Hiram Calvo, Irina Gelbukh, and Grigori Sidorov. 2024. Adaptation of transformer-based models for depression detection. Computación y Sistemas, 28(1). ↗
  3. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1–79. ↗
  4. Fernando Aguilar-Canto, Marco A Cardoso-Moreno, Diana Jiménez, and Hiram Calvo. 2023. Gpt-2 versus gpt-3 and bloom: Llms for llms generative text detection. In IberLEF@ SEPLN. find
  5. Omar García-Vázquez, Tania Alcántara, Hiram Calvo, and Grigori Sidorov. 2023. Llm’s for spanish song text analysis and classification using language variants. In Mexican International Conference on Artificial Intelligence, pages 118–127. Springer. find
  6. Abdullah Alshanqiti, Abdallah Namoun, Aeshah Alsughayyir, Aisha Mousa Mashraqi, Abdul Rehman Gilal, and Sami Saad Albouq. 2021. Leveraging distilbert for summarizing arabic text: an extractive dual-stage approach. IEEE Access, 9:135594–135607. ↗
  7. Syed Immamul Ansarullah, Mudasir Manzoor Kirmani, Sami Alshmrany, and Arfat Firdous. 2024. Ethical issues around artificial intelligence. In A Biologist s Ali Garib and Tina A Coffelt. 2024. Detecting the anomalies: Exploring implications of qualitative research in identifying ai-generated text for ai-assisted composition instruction. Computers and Composition, 73:102869. Thang Ta Hoang. The combination of bert and data oversampling for answer type prediction. ↗
  8. Slawomir Kierner, Jacek Kucharski, and Zofia Kierner. 2023. Taxonomy of hybrid architectures involving rule-based reasoning and machine learning in clinical decision systems: A scoping review. Journal of Biomedical Informatics, 144:104428. ↗
  9. Grigori Sidorov, Fazlourrahman Balouchzahi, Sabur Butt, and Alexander Gelbukh. 2023. Regret and hope on transformers: An analysis of transformers on regret and hope speech detection datasets. Applied Sciences, 13(6):3983. ↗
  10. Olga Kolesnikova, Mesay Gemeda Yigezu, Atnafu Lambebo Tonja, Michael Meles Woldeyohannis, Grigori Sidorov, and Alexander Gelbukh. 2023. Ginger disease detection using a computer vision pre-trained model. In Innovations in Machine and Deep Learning: Case Studies and Applications, pages 419–432. Springer. ↗
  11. Miguel Soto, Cesar Macias, Marco Cardoso-Moreno, Tania Alcántara, Omar García, and Hiram Calvo. 2024. Cognicic at emospeech-iberlef2024: Exploring multimodal emotion recognition in spanish: Deep learning approaches for speech-text analysis. find
  12. Mayte H Laureano and Hiram Calvo. 2024. Computational study of dream interpretations: Psychoanalytic human vs artificial analyses. In 2024 IEEE Congress on Evolutionary Computation (CEC), pages 1–9. IEEE. find
  13. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGE: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, Bangkok, Thailand. Association for Computational Linguistics. ↗
  14. Luciane Maimone and Jason Jolley. 2023. Looks like google to me: Instructor ability to detect machine translation in l2 spanish writing. Foreign Language Annals, 56(3):627–644. ↗
  15. Olumide E Ojo, Olaronke O Adebanji, Hiram Calvo, Damian O Dieke, Olumuyiwa E Ojo, Seye E Akin-sanya, Tolulope O Abiola, and Anna Feldman. 2023a. Legend at araieval shared task: Persuasion technique detection using a language-agnostic text representation model. In 1st Arabic Natural Language Processing Conference, ArabicNLP 2023, pages 594–599. Association for Computational Linguistics (ACL).
  16. Olumide E Ojo, Olaronke O Adebanji, Hiram Calvo, Alexander Gelbukh, Anna Feldman, and Ofir Ben Shoham. 2024. Doctor or ai? efficient neural network for response classification in health consultations. IEEE Access. ↗
  17. Olumide Ebenezer Ojo, Olaronke Oluwayemisi Adebanji, Hiram Calvo, Alexander Gelbukh, Anna Feldman, and Grigori Sidorov. 2023b. Hate and offensive content identification in indo-aryan languages using transformer-based models. ↗
  18. Olumide Ebenezer Ojo, Hoang Thang Ta, Alexander Gelbukh, Hiram Calvo, Olaronke Oluwayemisi Adebanji, and Grigori Sidorov. 2023c. Transformer-based approaches to sentiment detection. In Recent Developments and the New Directions of Research, Foundations, and Applications: Selected Papers of the 8th World Conference on Soft Computing, February 03–05, 2022, Baku, Azerbaijan, Vol. II, pages 101–110. Springer. ↗
  19. Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, Jinyan Su, Minh Ngoc Ta, Mervat Abassy, Kareem Elozeiri, Saad El Dine Ahmed, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Nurkhan Laiyk, Osama Mohammed Afzal, Ryuto Koike, Masahiro Kaneko, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2025. GenAI content detection task 1: English and multilingual machine-generated text detection: AI vs. human. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. ↗
  20. Mesay Gemeda Yigezu, Tadesse Kebede, Olga Kolesnikova, Grigori Sidorov, and Alexander Gelbukh. 2023. Habesha@ dravidianlangtech: Utilizing deep and transfer learning approaches for sentiment analysis. In Proceedings of the Third Workshop on Speech and Language Technologies for Dravidian Languages, pages 239–243. find
  21. Wenxin Zheng. 2024. Ai vs. human: A comparative study of cohesion and coherence in academic texts between human-written and chatgpt-generated texts. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abiola-2025-cic-nlp-genai-detection-task-2025-genaidetect-1-29/