Elamur Research⌕
☰
home / papers / 2025 / GenAI Content Detection Task 2: AI vs. Human – Academic Essay Authenticity Challenge

GenAI Content Detection Task 2: AI vs. Human – Academic Essay Authenticity Challenge

Paper · 2025 · Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu et al.

Summary

This overview reports on the first Academic Essay Authenticity Challenge, a shared task held within the GenAI Content Detection workshop at COLING 2025. Participants received an essay and had to decide whether a human or an LLM wrote it. The task was a binary classification problem in English and Arabic. The organisers built the human side from language-assessment collections such as IELTS, the ETS Corpus of Non-Native Written English and the CERCLL corpus. They kept only essays that were handwritten or written under classroom supervision, so that none could have had AI help. The machine side was generated with persona-conditioned prompts that vary study level, home country, proficiency and, for Arabic, nativity. Topics never repeat across splits. Systems were ranked by macro-F1. Most teams fine-tuned transformer encoders, and one team used Llama 2 and Llama 3. The best systems scored above 0.98 F1 in both languages, far above the n-gram baseline. The authors note that the small dataset, especially for Arabic, limits how far these numbers generalise.

Key points

  • 99 teams registered, and 56 of them participated across the development and evaluation stages. In the final evaluation, 25 teams submitted for English and 21 for Arabic, and five teams wrote system description papers.
  • Human data sources:
    • IELTS Writing Scored Essays Dataset: about 1,200 essays with examiner feedback and scores.
    • ETS Corpus of Non-Native Written English: 12,100 essays.
    • CERCLL corpus: roughly 270 Arabic essays by L2 and heritage speakers.
  • The data was released in two phases. The development phase provided train, validation and dev-test sets with topic-disjoint splits. The evaluation phase provided the final test set.
  • Top results (macro-F1):
    • English: CMI-AIGCX scored 0.999, using Llama 2/3 plus a fine-tuned XLM-roberta.
    • Arabic: IntegrityAI scored 0.984 with a fine-tuned Electra.
    • The n-gram baseline stayed below 0.5 in both languages.
  • Two of the described systems:
    • EssayDetect combined pretrained embeddings with stylometric and linguistic features, used LIME, focal loss and layer-wise freezing.
    • Team RA found that AraBERT gave the best F1 across both languages.

Interesting ideas

  • I1. The organisers first grouped topics by theme, then gave some topics only to LLM generation and kept the rest only for human essays, with no topic shared across splits; this stops a detector from scoring well just by recognising topics.
  • I2. Generation prompts were filled from a grid of personas (study level, country, proficiency, nativity), giving machine texts that imitate many kinds of writers instead of a single LLM voice.
  • I3. Human essays were accepted only when their origin was verifiable (handwritten or supervised), so the human class is trustworthy.
  • I4. EssayDetect froze the lower layers during fine-tuning so they kept general linguistic knowledge, while the upper layers learned task-specific style cues.
  • I5. EssayDetect also used LIME to show which features drove each decision, giving an explanation that can be read alongside the label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets for Mova readers could hold out whole themes or story families, so a score drop on unseen themes works as a negative control against memorisation.
  • I2 (learn): LLM teacher tiers could be asked to produce texts across a controlled grid of writer profiles, which would make the silver data for parser and frame training more diverse.
  • I3 (learn): Gold evaluation material could be tagged by provenance, so pure human text is kept apart from text that may contain LLM output before witness parsers are scored.
  • I4 (tree): gparse could be adapted to a new UD dialect or treebank by freezing its lower transformer layers, which may reduce forgetting when the new data is small.
  • I5 (answer): A similar attribution pass over the features a Mova reader uses could check that answers rest on frame and world evidence rather than surface cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

Also on the byline: TOBB University of Economics and Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keleş, Fatema Ahmad, Tasnim Mohiuddin, George Mikros and Firoj Alam. 2025. GenAI Content Detection Task 2: AI vs. Human – Academic Essay Authenticity Challenge. Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chowdhury2025genai,
  title = {{GenAI Content Detection Task 2: AI vs. Human – Academic Essay Authenticity Challenge}},
  author = {Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keleş, Fatema Ahmad, Tasnim Mohiuddin, George Mikros and Firoj Alam},
  booktitle = {Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect)},
  year = {2025},
  url = {https://aclanthology.org/2025.genaidetect-1.37/}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. ↗
  2. Shifali Agrahari, Subhashi Jayant, Saurabh Kumar, and Ranbir Sanasam. 2025. Team EssayDetect at GenAI Detection Task 2: Guardians of Academic Integrity: Multilingual Detection of AI-Generated Essays. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. find
  3. Abdelhamid M. Ahmed, Xiao Zhang, Lameya M. Rezk, and Wajdi Zaghouani. 2023. Building an annotated l1 arabic/l2 english bilingual writer corpus: The qatari corpus of argumentative writing (qcaw). Corpus-based Studies across Humanities, 1(1):183–215. ↗
  4. Mohammad AL-Smadi. 2025. IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELEC-TRA and Stylometry. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), AYG Alfaifi and ES Atwell. 2013. Arabic learner corpus v1: A new resource for arabic language research. In Second Workshop on Arabic Corpus Linguistics. find
  5. Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. 2023. ConDA: Contrastive domain adaptation for AI-generated text detection. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 598–610, Nusa Dua, Bali. Association for Computational Linguistics. ↗
  6. Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. 2024. Genqa: Generating millions of instructions from a handful of prompts. Preprint, arXiv:2406.10323. ↗
  7. Kohinoor Monish Darda, Marion Carre, and Emily S. Cross. 2023. Value attributed to text-based archives generated by artificial intelligence. Royal Society Open Science, 10(2):220915. ↗
  8. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12463– 12492, Bangkok, Thailand. Association for Computational Linguistics. ↗
  9. Matthias Gallé, Jos Rozen, Germán Kruszewski, and Hady Elsahar. 2021. Unsupervised and distributional detection of machine-generated text. arXiv preprint arXiv:2111.02878. ↗
  10. Rana Gharib and Ahmed Elgendy. 2025. RA at GenAI Detection Task 2: Fine-tuned Language Models For Detection of Academic Authenticity, Results and Thoughts. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. find
  11. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. ↗
  12. Hans WA Hanley and Zakir Durumeric. 2024. Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 542–556. find
  13. Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. 2022. Protecting intellectual property of language generation apis with lexical watermark. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10758–10766. find
  14. Vijayasaradhi Indurthi and Vasudeva Varma. 2025. Tesla at GenAI Content Detection Task 1: LLM Agents in Multilingual Machine-Generated Text Detection. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. find
  15. Jiao Kaijie, Yao Xingyu, Ma Shixuan, Fang Sifan, Guo Zikang, Xu Benfeng, Zhang Licheng, Wang Quan, Zhang Yongdong, and Mao Zhendong. 2025. CMI-AIGCX at GenAI Detection Task 2: Leveraging Multilingual Proxy LLMs for Machine-Generated Text Detection in Academic Essays. In Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE. International Conference on Computational Linguistics. find
  16. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR. ↗
  17. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. GPT detectors are biased against non-native english writers. Patterns, 4(7):100779. ↗
  18. George Mikros, Athanasios Koursaris, Dimitrios Bil-ianos, and George Markopoulos. 2023. AI-writing detection using an ensemble of transformers and stylometric features. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023) co-located with the Conference of the Spanish Society for Natural Language Processing (SEPLN 2023), volume 3496 of CEUR Workshop Proceedings, pages 1–14, Jaén, Spain. find
  19. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. OpenAI Blog. find
  20. Mike Perkins, Jasper Roe, Binh H. Vu, Darius Postma, Don Hickerson, James McGaughran, and Huy Q. Khuat. 2024. Simple techniques to bypass GenAI text detectors: Implications for inclusive education. International Journal of Educational Technology in Higher Education, 21(1):53. ↗
  21. Hooman H. Rashidi, Brandon D. Fennell, Samer Al-bahra, Bo Hu, and Tom Gorbett. 2023. The chatgpt conundrum: Human-generated scientific manuscripts misidentified as ai creations by ai text detection tool. Journal of Pathology Informatics, 14:100342. ↗
  22. Aditya Shah, Prateek Ranka, Urmi Dedhia, Shruti Prasad, Siddhi Muni, and Kiran Bhowmick. 2023. Detecting and unmasking AI-generated texts through explainable artificial intelligence using stylistic features. International Journal of Advanced Computer Science and Applications, 14(10):1–10. ↗
  23. Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. 2021. Dawn: Dynamic adversarial watermarking of neural networks. In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, page 4417–4425, New York, NY, USA. Association for Computing Machinery. find
  24. Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The science of detecting llm-generated text. Communications of the ACM, 67(4):50–59. ↗
  25. Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. ↗
  26. Vismay Vora, Jenil Savla, Deevya Mehta, and Aruna Gawade. 2023. A multimodal approach for detecting AI generated content using BERT and CNN. International Journal on Recent and Innovation Trends in Computing and Communication, 11(9):691–701. ↗
  27. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Thomas Arnold, Chenxi Whitehouse, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024a. SemEval-2024 Task 8: Multidomain, multimodel and multilingual machine-generated text detection. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 2057–2079, Mexico City, Mexico. Association for Computational Linguistics. ↗
  28. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4: Multi-generator, multi-domain, and multilingual black-box machine-generated text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1369– 1407, St. Julian’s, Malta. Association for Computational Linguistics. find
  29. Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Guerrero-Dib, Olumide Popoola, Petr Šigut, and Lorna Waddington. 2023. Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1):1–39. ↗
  30. Hongyu Wu and Tom Flanagan. 2023. The limits of AI content detectors. Journal of Student Research, 12(3):1–7. ↗
  31. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724. ↗
  32. Peipeng Yu, Jiahan Chen, Xuan Feng, and Zhihua Xia. 2023. Cheat: A large-scale dataset for detecting chatgpt-written abstracts. arXiv preprint arXiv:2304.12008. ↗
  33. Wajdi Zaghouani, Abdelhamid Ahmed, Xiao Zhang, and Lameya Rezk. 2024. QCAW 1.0: Building a qatari corpus of student argumentative writing. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13382–13394, Torino, Italia. ELRA and ICCL.
  34. Wataru Zaitsu and Mingzhe Jin. 2023. Distinguishing chatgpt(-3.5, -4)-generated and human-written papers through japanese stylometric analysis. PLOS ONE, 18(8):1–12. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2025-genai-content-detection-task-2/