Elamur Research⌕
☰
home / papers / 2025 / The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text

The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text

Summary

The paper asks whether text written by large language models in Arabic carries recognisable traces, and whether those traces can drive a reliable detector. The authors built two corpora. One holds Arabic academic abstracts scraped from the Algerian Scientific Journals database. The other holds informal social media reviews. Machine counterparts came from ALLaM, Jais, Llama 3.1 and GPT-4, using three prompting regimes: writing from the title alone, writing with access to paper content, and polishing an existing human text. This yielded about 11.7k generated samples.

A stylometric comparison of word-frequency behaviour, length and semantic measures shows systematic gaps between human and model writing. Fine-tuned BERT-based classifiers then separate the two almost perfectly on scholarly prose, and the academic detectors also transfer well to other models. On social media, detection is weaker and transfer can collapse. The work matters because it is the broadest study so far of AI-text detection for a lower-resource language. It also shows how much detectability depends on genre.

Key points

  • Human abstracts were restricted to papers published 2010–2022, so that no LLM-written text could slip into the "human" side. Over 60,000 papers were scraped in total.
  • Detection on academic abstracts reached 99.9% F1 for GPT-4 output and 99.69% for Jais. Cross-model F1 on abstracts rarely fell under 90%.
  • On social media, detectors scored 90.69–98.66% F1 on texts from the model they were trained on. Transfer across models could be very poor: a detector trained on Llama dropped to 29.25% F1 on OpenAI text, and the reverse direction also failed. Precision stayed high while recall fell.
  • In the five-way classification (human plus four models) on social media, Llama was easiest to identify (95.37% F1) and ALLaM hardest (66.51%). ALLaM was often mistaken for OpenAI, Jais or human text.
  • Compared with human writing, the models' academic text loses its rare words faster and overuses common ones. Their social media posts were only 25–72% of human length but kept the meaning. Code is released at KFUPM-JRCAI/arabic-text-detection.

Interesting ideas

  • I1. Using a publication-date cutoff that predates LLMs gives a human corpus that is guaranteed clean, which makes it a trustworthy negative class. — For Mova (learn): Pre-2022 text could serve as a fixed negative control when checking whether silver data or LLM-teacher outputs drift away from natural human syntax.
  • I2. Generating several machine versions from the same human seed (title only, with content, refinement) produces paired texts that differ in how much the model invents versus edits. — For Mova (learn): Seeding LLM teacher tiers with the same source sentence at different freedom levels would show which parser errors come from rewriting style rather than from the content itself.
  • I3. A full train-model × test-model matrix that reports precision and recall separately shows that a detector failing to transfer usually misses examples rather than raising false alarms. — For Mova (learn): Witness parsers could be scored the same way, each against the others' silver output, with recall and precision split to locate one-directional blind spots.
  • I4. Comparing the shape of the log-log frequency curve, especially how quickly rare words disappear, works as a cheap fingerprint of non-human text. — For Mova (back): Text produced by the text → graph → text compressor could be checked against the human rare-word tail to detect flattened, generic vocabulary in generation.
  • I5. The finding that models compress informal posts heavily while preserving meaning separates length from semantic fidelity as two distinct measures. — For Mova (back): Regenerated text could be evaluated on its length ratio and its meaning-graph overlap separately, so that brevity is not confused with information loss.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.23276 · arXiv 2505.23276

@misc{alshaibani2025arabic,
  title = {{The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text}},
  author = {Al-Shaibani, Maged S. and Ahmed, Moataz},
  year = {2025},
  doi = {10.48550/arxiv.2505.23276},
  eprint = {2505.23276},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.23276}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. B. Brown, Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020). ↗
  2. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023). ↗
  3. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al., Gpt-neox-20b: An open-source autoregressive language model, arXiv preprint arXiv:2204.06745 (2022). ↗ our card
  4. S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, et al., Textbooks are all you need, arXiv preprint arXiv:2306.11644 (2023). ↗
  5. Cohere For AI, c4ai-command-r-v01 (revision 8089a08), 2024. URL: https://huggingface.co/CohereForAI/c4ai-command-r-v01. doi:10.57967/hf/3139. ↗
  6. E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machine-generated text detection using probability curvature, in: International Conference on Machine Learning, PMLR, 2023, pp. 24950–24962. find our card
  7. G. Bao, Y. Zhao, Z. Teng, L. Yang, Y. Zhang, Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature, arXiv preprint arXiv:2310.05130 (2023). ↗ our card
  8. X. Yang, L. Pan, X. Zhao, H. Chen, L. Petzold, W. Y. Wang, W. Cheng, A survey on detection of llms-generated content, arXiv preprint arXiv:2310.15654 (2023). ↗
  9. P. Yu, J. Chen, X. Feng, Z. Xia, Cheat: A large-scale dataset for detecting chatgpt-written abstracts, arXiv preprint arXiv:2304.12008 (2023). ↗
  10. W. Liang, Y. Zhang, Z. Wu, H. Lepp, W. Ji, X. Zhao, H. Cao, S. Liu, S. He, Z. Huang, et al., Mapping the increasing use of llms in scientific papers, arXiv preprint arXiv:2404.01268 (2024). ↗
  11. A. Muñoz-Ortiz, C. Gómez-Rodríguez, D. Vilares, Contrasting linguistic patterns in human and llm-generated news text, Artificial Intelligence Review 57 (2024) 265. find
  12. N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, W. Marshall, G. Gosal, C. Liu, Z. Chen, et al., Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models, arXiv preprint arXiv:2308.16149 (2023). ↗ our card
  13. H. Huang, F. Yu, J. Zhu, X. Sun, H. Cheng, D. Song, Z. Chen, A. Alharthi, B. An, J. He, et al., Acegpt, localizing large language models in arabic, arXiv preprint arXiv:2309.12053 (2023). ↗ our card
  14. M. S. Bari, Y. Alnumay, N. A. Alzahrani, N. M. Alotaibi, H. A. Alyahya, S. AlRashed, F. A. Mirza, S. Z. Alsubaie, H. A. Alahmed, G. Alabduljabbar, et al., ALLaM: Large language models for arabic and english, arXiv preprint arXiv:2407.15390 (2024). ↗ our card
  15. H. Alshammari, A. El-Sayed, Airabic: Arabic dataset for performance evaluation of ai detectors, in: 2023 International Conference on Machine Learning and Applications (ICMLA), IEEE, 2023, pp. 864–870. find
  16. H. Alshammari, A. El-Sayed, K. Elleithy, Ai-generated text detector for arabic language using encoder-based transformer architecture, Big Data and Cognitive Computing 8 (2024) 32. find
  17. J. Wu, S. Yang, R. Zhan, Y. Yuan, D. F. Wong, L. S. Chao, A survey on llm-gernerated text detection: Necessity, methods, and future directions, arXiv preprint arXiv:2310.14724 (2023). ↗
  18. R. Koike, M. Kaneko, N. Okazaki, Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 21258–21266. find
  19. Z. Yang, Z. Feng, R. Huo, H. Lin, H. Zheng, R. Nie, H. Chen, The imitation game revisited: A comprehensive survey on recent advances in ai-generated text detection, Expert Systems with Applications 272 (2025) 126694. find
  20. X. Hu, W. Ou, S. Acharya, S. H. Ding, R. D’Gama, H. Yu, Tdrlm: Stylometric learning for authorship verification by topic-debiasing, Expert Systems with Applications 233 (2023) 120745. find
  21. R. Ramezani, A language-independent authorship attribution approach for author identification of text documents, Expert Systems with Applications 180 (2021) 115139. find
  22. C. Opara, P. Modesti, L. Golightly, Evaluating spam filters and stylometric detection of ai-generated phishing emails, Expert Systems with Applications 276 (2025) 127044. find
  23. S. Herbold, A. Hautli-Janisz, U. Heuer, Z. Kikteva, A. Trautsch, A large-scale comparison of human-written versus chatgpt-generated essays, Scientific Reports 13 (2023) 1–11. find
  24. T. Kumarage, J. Garland, A. Bhattacharjee, K. Trapeznikov, S. Ruston, H. Liu, Stylometric detection of ai-generated text in twitter timelines, arXiv preprint arXiv:2303.03697 (2023). ↗
  25. S. Feng, R. Banerjee, Y. Choi, Syntactic stylometry for deception detection, in: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, 2012, pp. 171–175. find
  26. S. Takahashi, K. Tanaka-Ishii, Evaluating computational language models with scaling properties of natural language, Computational Linguistics 45 (2019) 481–513. find
  27. B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, Y. Wu, How close is chatgpt to human experts? comparison corpus, evaluation, and detection, arXiv preprint arxiv:2301.07597 (2023). ↗ our card
  28. Z. Su, X. Wu, W. Zhou, G. Ma, S. Hu, Hc3 plus: A semantic-invariant human chatgpt comparison corpus, arXiv preprint arXiv:2309.02731 (2023). ↗
  29. Y. Li, Q. Li, L. Cui, W. Bi, Z. Wang, L. Wang, L. Yang, S. Shi, Y. Zhang, Mage: Machine-generated text detection in the wild, arXiv preprint arXiv:2305.13242 (2023). ↗
  30. Ai detector - the original ai checker for chatgpt & more, https://gptzero.me/, ???? (Accessed on 12/04/2024). ↗
  31. W. Antoun, F. Baly, H. Hajj, Araelectra: Pre-training text discriminators for arabic language understanding, arXiv preprint arXiv:2012.15516 (2020). ↗
  32. A. Conneau, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019). ↗
  33. A. Elnagar, O. Einea, Brad 1.0: Book reviews in arabic dataset, in: 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), IEEE, 2016, pp. 1–8. find
  34. A. Elnagar, Y. S. Khalifa, A. Einea, Hotel arabic-reviews dataset construction for sentiment analysis applications, Intelligent natural language processing: Trends and applications (2018) 35–52. find
  35. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al., The llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024). ↗
  36. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023). ↗ our card
  37. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318. find our card
  38. S. Banerjee, A. Lavie, Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72. find our card
  39. C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81. find
  40. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, arXiv preprint arXiv:1904.09675 (2019). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-2025-shaibani-arabic-ai-fingerprint-stylometric-analysis/