Elamur Research⌕
☰
home / papers / 2025 / ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection

ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection

Summary

Arabic has had no large public resource for telling human-written text apart from text produced by LLMs. Earlier collections were small, used a single generator, or covered only one or two genres, so it was hard to test whether detectors generalize. The authors built ALHD to fill this gap. It is a balanced corpus of more than 400K samples across news, social media and reviews, covering both MSA and dialectal Arabic. The human side comes from six existing Arabic datasets. The machine side was produced by three commercial LLMs. The release includes preprocessing, annotations and fixed splits.

The authors benchmarked classical classifiers, fine-tuned BERT-style models, and prompted LLMs in zero-shot and few-shot modes. Fine-tuned BERT models came out on top, often above 90% accuracy. Classical models formed a weaker but respectable baseline, and prompted LLMs did worst. All model families degraded when tested on a genre they had not been trained on. News was the hardest genre, because generated articles closely mimic human journalistic style. ALHD therefore gives Arabic detection research a shared testbed focused on robustness.

Key points

  • ALHD has over 400K balanced human/LLM samples in three genres (news, social media, reviews), covering MSA and some dialectal Arabic.
  • The human texts come from SANAD, ANAD, MDAT, MASC, HARD and BRAD.
  • Generated texts came from GPT-3.5-turbo, Gemini 2.5 and Command-R, called through official APIs with default settings.
  • Fine-tuned BERT models beat traditional ML and zero-/few-shot LLM prompting.
  • Cross-genre transfer is weak, especially when news is the target genre.
  • The authors ran 540 reproducible experiments.
  • The data is on Zenodo under CC BY-NC 4.0, and the code is on GitHub under MIT.

Interesting ideas

  • I1. Train on one genre and test on another, so that apparent in-genre accuracy cannot hide a model that has only learned genre-specific cues. — For Mova (learn): Evaluate the parser and frame extractor on held-out genres (e.g., train on news-like text, test on fables) to measure whether rules and models truly generalize.
  • I2. Pair existing human-only corpora with matched machine-generated counterparts instead of collecting everything from scratch, keeping classes balanced per source. — For Mova (learn): Build balanced silver sets from existing treebank sentences plus LLM-written analogues, as a negative control for whether witness parsers favour machine-style text.
  • I3. Generate machine text with default API parameters, so the data reflects ordinary user behaviour rather than a tuned adversary. — For Mova (learn): When using LLM teacher tiers to produce training text, record and fix the generation settings so the resulting data distribution is known and reproducible.
  • I4. The finding that news is hardest suggests surface style alone is a fragile signal once generators imitate a register well. — For Mova (back): Rather than relying on style, check regenerated text against the meaning graph (who did what to whom) to judge fidelity.
  • I5. Compare cheap classical baselines, fine-tuned encoders and prompted LLMs side by side under identical splits, so the relative value of each tier is clear. — For Mova (learn): Benchmark Mova's explainable pipeline against a fine-tuned encoder and a prompted LLM on the same school-test splits, so its gains or gaps are measured honestly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.03502 · arXiv 2510.03502

@misc{khairallah2025alhd,
  title = {{ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection}},
  author = {Khairallah, Ali and Zubiaga, Arkaitz},
  year = {2025},
  doi = {10.48550/arxiv.2510.03502},
  eprint = {2510.03502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.03502}
}

References

57 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models, ArXiv preprint abs/2303.18223 (2023). URL https://arxiv.org/abs/2303.18223 ↗ our card
  2. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, J. Gao, Large language models: A survey, ArXiv preprint abs/2402.06196 (2024). URL https://arxiv.org/abs/2402.06196 ↗ our card
  3. A. Zubiaga, Natural language processing in the era of large language models, Frontiers in artificial intelligence 6 (2024) 1350306. find
  4. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, Y. Choi, Defending against neural fake news, in: H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 9051–9062. URL https://proceedings.neurips.cc/paper/2019/hash/3e9f0fc9b2f89e043bc6233994dfcf76-Abstract.html ↗ our card
  5. H. Wang, W. Fu, Y. Tang, Z. Chen, Y. Huang, J. Piao, C. Gao, F. Xu, T. Jiang, Y. Li, A survey on responsible llms: Inherent risk, malicious use, and mitigation strategy, ArXiv preprint abs/2501.09431 (2025). URL https://arxiv.org/abs/2501.09431 ↗
  6. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html ↗
  7. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, I. Gabriel, Ethical and social risks of harm from language models, arXiv preprint (2021). arXiv:2112.04359. ↗ our card
  8. H. Alshammari, A. El-Sayed, K. Elleithy, Ai-generated text detector for arabic language using encoder-based transformer architecture, Big Data and Cognitive Computing 8 (3) (2024) 32. find
  9. W. Antoun, F. Baly, H. Hajj, AraBERT: Transformer-based model for Arabic language understanding, in: H. Al-Khalifa, W. Magdy, K. Darwish, T. Elsayed, H. Mubarak (Eds.), Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, European Language Resource Association, Marseille, France, 2020, pp. 9–15. URL https://aclanthology.org/2020.osact-1.2 ↗ our card
  10. A. Boutadjine, F. Harrag, K. Shaalan, Human vs. machine: A comparative study on the detection of ai-generated content, ACM Transactions on Asian and Low-Resource Language Information Processing 24 (2) (2025) 1–26. find
  11. A. Abdelali, H. Mubarak, S. Chowdhury, M. Hasanain, B. Mousi, S. Boughorbel, S. Abdaljalil, Y. El Kheir, D. Izham, F. Dalvi, M. Hawasly, N. Nazar, Y. Elshahawy, A. Ali, N. Durrani, N. Milic-Frayling, F. Alam, LAraBench: Benchmarking Arabic AI with large language models, in: Y. Graham, M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, St. Julian’s, Malta, 2024, pp. 487–520. URL https://aclanthology.org/2024.eacl-long.30 ↗ our card
  12. J. Wu, S. Yang, R. Zhan, Y. Yuan, L. S. Chao, D. F. Wong, A survey on llm-generated text detection: Necessity, methods, and future directions, Computational Linguistics (2025) 1–66. find our card
  13. D. Valiaiev, Detection of machine-generated text: Literature survey, arXiv preprint (2024). arXiv:2402.01642. ↗
  14. B. Xu, A. Zubiaga, Understanding the effects of rlhf on the quality and detectability of llm-generated texts, arXiv preprint (2025). arXiv:2503.17965. ↗
  15. X. Yang, K. Chen, W. Zhang, C. Liu, Y. Qi, J. Zhang, H. Fang, N. Yu, Watermarking text generated by black-box language models, arXiv preprint (2023). arXiv:2305.08883. ↗
  16. H. T. Lau, A. Zubiaga, Understanding the effects of human-written paraphrases in llm-generated text detection, Natural Language Processing Journal (2025) 100151. find
  17. M. S. Al-Shaibani, M. Ahmed, The arabic ai fingerprint: Stylometric analysis and detection of large language models text, arXiv preprint (2025). arXiv:2505.23276. ↗ our card
  18. M. Abdul-Mageed, A. Elmadany, E. M. B. Nagoudi, ARBERT & MAR-BERT: Deep bidirectional transformers for Arabic, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 7088–7105. doi:10.18653/v1/2021.acl-long.551. URL https://aclanthology.org/2021.acl-long.551 ↗
  19. E. Boujou, H. Chataoui, A. E. Mekki, S. Benjelloun, I. Chairi, I. Berrada, An open access nlp dataset for arabic dialects: Data collection, labeling, and model construction, arXiv preprint (2021). arXiv:2102.11000. ↗ our card
  20. A. Khalil, M. Jarrah, M. Aldwairi, M. Jaradat, Afnd: Arabic fake news dataset for the detection and classification of articles credibility, Data in Brief 42 (2022) 108141. find
  21. X. Duan, B. Xiao, X. Tang, Z. G. Cai, Hlb: Benchmarking llms’ humanlikeness in language use, ArXiv preprint abs/2409.15890 (2024). URL https://arxiv.org/abs/2409.15890 ↗
  22. L. Fröhling, A. Zubiaga, Feature-based detection of automated language models: tackling gpt-2, gpt-3 and grover, PeerJ Computer Science 7 (2021) e443. find
  23. M. A. Bagwell, Organizational decisions about cyber security in small to mid-sized businesses: A qualitative study, Ph.D. thesis, Northcentral University (2016). find
  24. A.-P. Working, Group (apwg), “phishing activity trends report, 3rd quarter 2024, ” apwg, https://apwg.org/trendsreports/apwg_trends_report_q3_2024.pdf. Tech. Rep (2024). ↗
  25. Federal Bureau of Investigation, Internet crime report 2023, retrieved 2024-07-01. https://www.ic3.gov/Media/PDF/AnnualReport/2023_IC3Report.pdf (2024). ↗
  26. A. Gutub, Emerging arabic text watermarking utilizing combinations of different diacritics, Arabian Journal for Science and Engineering 49 (9) (2024) 12317–12332. find
  27. R. A. Alotaibi, L. A. Elrefaei, Arabic text watermarking: A review, International Journal of Artificial Intelligence & Applications 6 (4) (2015) 01–16. find
  28. M. Abbache, A. Abbache, J. Xu, F. Meziane, X. Wen, The impact of arabic diacritization on word embeddings, ACM Transactions on Asian and Low-Resource Language Information Processing 22 (6) (2023) 1–30. find
  29. S. M. Abdulmaged, Identification of tampering in arabic text documents using zero-watermarking algorithm, Journal La Multiapp 5 (5) (2024) 680–691. find
  30. H. Alshammari, E.-S. Ahmed, Airabic: Arabic dataset for performance evaluation of ai detectors, in: Proc. of the 2023 Int. Conf. on Machine Learning and Applications (ICMLA), Jacksonville, FL, USA, 2023, pp. 864–870. find
  31. F. Harrag, M. Dabbah, K. Darwish, A. Abdelali, Bert transformer model for detecting Arabic GPT2 auto-generated tweets, in: I. Zitouni, M. Abdul-Mageed, H. Bouamor, F. Bougares, M. El-Haj, N. Tomeh, W. Zaghouani (Eds.), Proceedings of the Fifth Arabic Natural Language Processing Workshop, Association for Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 207–214. URL https://aclanthology.org/2020.wanlp-1.19 ↗
  32. H. Himdi, N. Zamzami, F. Najar, M. Alrehaili, N. Bouguila, Arabic fake news dataset development: Humans and ai-generated contributions, IEEE Access 13 (2025) 62234–62253, doi:10.1109/AC-CESS.2025.3556376. ↗
  33. M. Altamimi, A. M. Alayba, Anad: Arabic news article dataset, Data in Brief 50 (2023) 109460. find
  34. O. Einea, A. Elnagar, R. Al Debsi, Sanad: Single-label arabic news articles dataset for automatic text categorization, Data in Brief 25 (2019) 104076. find
  35. A. Elnagar, O. Einea, Brad 1.0: Book reviews in arabic dataset, in: 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), 2016, pp. 1–8. find
  36. A. Elnagar, Y. S. Khalifa, A. Einea, Hotel arabic-reviews dataset construction for sentiment analysis applications, Intelligent Natural Language Processing: Trends and Applications (2018) 35–52. find
  37. M. Al-Fetyani, M. Al-Barham, G. Abandah, A. Alsharkawi, M. Dawas, Masc: Massive arabic speech corpus, in: 2022 IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 1006–1013. find
  38. D. Ippolito, D. Duckworth, C. Callison-Burch, D. Eck, Automatic detection of generated text is easiest when humans are fooled, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 1808–1822. doi:10.18653/v1/2020.acl-main.164. URL https://aclanthology.org/2020.acl-main.164 ↗ our card
  39. G. Jawahar, M. Abdul-Mageed, L. Lakshmanan, V.S., Automatic detection of machine generated text: A critical survey, in: D. Scott, N. Bel, C. Zong (Eds.), Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 2296–2309. doi:10.18653/v1/2020.coling-main.208. URL https://aclanthology.org/2020.coling-main.208 ↗ our card
  40. M. Abdul-Mageed, A. Elmadany, E. M. B. Nagoudi, ARBERT & MAR-BERT: Deep bidirectional transformers for Arabic, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 7088–7105. doi:10.18653/v1/2021.acl-long.551. URL https://aclanthology.org/2021.acl-long.551 ↗
  41. W. Antoun, F. Baly, H. Hajj, AraELECTRA: Pre-training text discriminators for Arabic language understanding, in: N. Habash, H. Bouamor, H. Hajj, W. Magdy, W. Zaghouani, F. Bougares, N. Tomeh, I. Abu Farha, S. Touileb (Eds.), Proceedings of the Sixth Arabic Natural Language Processing Workshop, Association for Computational Linguistics, Kyiv, Ukraine (Virtual), 2021, pp. 191–195. URL https://aclanthology.org/2021.wanlp-1.20 ↗
  42. H. U. Khan, A. Naz, F. K. Alarfaj, N. Almusallam, Identifying artificial intelligence-generated content using the distilbert transformer and nlp techniques, Scientific Reports 15 (2025) 20366. find
  43. A. S. Alammary, Investigating the impact of pretraining corpora on the performance of arabic bert models, The Journal of Supercomputing 81 (1) (2025) 187. find
  44. Z. Jalil, A. M. Mirza, T. Iqbal, A zero-watermarking algorithm for text documents based on structural components, in: 2010 International Conference on Information and Emerging Technologies, 2010, pp. 1–5. find
  45. E. M. Ahmadoh, A. A.-A. Gutub, Utilization of two diacritics for arabic text steganography to enhance performance 3 (1) (2015). find
  46. F. N. Al-Wesabi, H. G. Iskandar, S. Alzahrani, A. Abdelmaboud, M. Abdul, N. Nemri, M. Medani, M. Y. Alghamdi, Arabic feature-based text watermarking technique for sensitive detecting tampering attack, Computers, Materials & Continua 68 (3) (2021). find
  47. OpenAI, Gpt-3.5-turbo, https://platform.openai.com/docs/models/gpt-3.5-turbo (2023). ↗
  48. Google, Gemini 2.5 flash, https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash (2024). ↗
  49. Cohere, (2024). Command-r, https://docs.cohere.com/v1/docs/command-r ↗
  50. T. King, S. Butcher, L. Zalewski, Apocrita – high performance computing cluster for queen mary university of london, doi:10.5281/zen-odo.438045. (2017). ↗
  51. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, D. Duchesnay, Scikit-learn: Machine learning in python, Journal of Machine Learning Research 12 (2011) 2825–2830. find
  52. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T. Liu, Lightgbm: A highly efficient gradient boosting decision tree, in: I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 3146–3154. URL https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html ↗
  53. A. Elmadany, E. B. Nagoudi, M. Abdul-Mageed, ORCA: A challenging benchmark for Arabic language understanding, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 9559–9586. doi:10.18653/v1/2023.findings-acl.609. URL https://aclanthology.org/2023.findings-acl.609 ↗
  54. A. Safaya, M. Abdullatif, D. Yuret, KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media, in: A. Herbelot, X. Zhu, A. Palmer, N. Schneider, J. May, E. Shutova (Eds.), Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 2054–2059. doi:10.18653/v1/2020.semeval-1.271. URL https://aclanthology.org/2020.semeval-1.271 ↗ our card
  55. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 ↗
  56. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 8440–8451. doi:10.18653/v1/2020.acl-main.747. URL https://aclanthology.org/2020.acl-main.747 ↗
  57. A. Khairallah, A. Zubiaga, ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection (2025). doi:10.5281/zenodo.17249602. URL https://doi.org/10.5281/zenodo.17249602 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khairallah-2025-alhd-large-scale-multigenre-benchmark/