Elamur Research⌕
☰
home / papers / 2025 / AI Text Detectors and the Misclassification of Slightly Polished Arabic Text

AI Text Detectors and the Misclassification of Slightly Polished Arabic Text

Summary

The paper examines whether AI-text detectors wrongly flag Arabic human writing as machine-generated after an LLM lightly edits it. The authors built two resources. The first is a balanced set of 800 Arabic articles, half human-written and half AI-generated, drawn from news and blog sources. They used it to screen 14 detectors: 10 LLMs prompted to act as judges and 4 commercial tools. The eight strongest were kept. The second resource is Ar-APT: 400 human articles, each rewritten by 10 LLMs at four requested edit levels (10%, 25%, 50%, 75%), for 16,400 samples in total. Meaning preservation was checked with cosine and Jaccard similarity against the originals.

Every retained detector labelled many of the polished human texts as AI-written. Claude-4 Sonnet was the best LLM judge at 83.51% but fell to 57.63% on LLaMA-3 edits. Originality.AI was the strongest commercial tool but dropped to 12% on Mistral or Gemma-3 edits. The work matters because these errors amount to false plagiarism accusations, and Arabic had no such study before.

Key points

  • The first dataset has 800 Arabic articles (400 human, 400 AI). Claude-4 Sonnet was the top LLM detector, with 83.51% accuracy and a 16.49% false-positive rate. Originality.AI was the top commercial detector, reported at 92% in the abstract and at 96% accuracy with 8% FPR elsewhere.
  • Ar-APT contains 400 originals plus 16,000 polished versions from 10 LLMs, including GPT-4o, GPT-3.5, Claude-4 Sonnet, LLaMA-3 70B, LLaMA-4 17B, Deepseek 3.1, Qwen-3, Mistral, Gemma-3 and Kimi K2. It supported 320 detection experiments, and the data is on GitHub.
  • At the 10% setting, Claude-4 Sonnet's accuracy on polished human text fell to 65.03% for Mistral edits, 65.71% for Gemma-3 edits and 57.63% for LLaMA-3 edits. For Originality.AI, the false-positive rate reached 88% on Mistral and Gemma-3 edits.
  • ZeroGPT reached about 80% accuracy with 38% FPR on the first set. On polished text its results barely moved across edit levels, for example a constant 57% for Claude-edited articles.
  • Claude-4 Sonnet was the best at preserving meaning, followed by Deepseek-3.1 and GPT-4o. Qwen-3 was the only polisher that handled Arabic poorly. GPT-3.5 made nearly identical edits regardless of the requested level.

Interesting ideas

  • I1. The authors took one original text and applied graded, meaning-preserving rewrites to it, then measured how far the model's decision drifted; this isolates sensitivity to surface form from sensitivity to content. — For Mova (learn): Use LLM-teacher paraphrases at several edit levels as negative controls, and require that the extracted frames and world state stay the same; any drift points to analysis that depends on wording rather than meaning.
  • I2. Before trusting the perturbations, they checked them with two lenses, lexical overlap (Jaccard) and semantic closeness (cosine), which revealed that some rewriters ignored the requested edit degree. — For Mova (back): Score the text → graph → text compressor on both axes, so that low word overlap combined with an identical meaning graph counts as success, while a high-overlap but unfaithful output is caught.
  • I3. Results were broken down by which model performed the rewrite, and that choice mattered far more than the edit percentage, exposing detector blind spots for rewriters underrepresented in training. — For Mova (tree): Evaluate the perceptron parser and gparse on sentences rewritten by different LLMs and report LAS per rewriter, to find stylistic sources the treebanks do not cover.
  • I4. The paper treats a false accusation as costlier than a missed detection and reports FPR separately rather than hiding it inside accuracy. — For Mova (answer): Track confidently wrong answers separately from abstentions on school tests and fable QA, and tune readers to prefer "unknown" over a wrong claim.
  • I5. The study used a two-stage evaluation: a cheap balanced screen first, then an expensive stress test run only on the survivors. — For Mova (learn): Screen candidate witness parsers or LLM teacher tiers on a small gold set before spending compute on large silver-data agreement runs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.16690 · arXiv 2511.16690

@misc{almohaimeed2025ai,
  title = {{AI Text Detectors and the Misclassification of Slightly Polished Arabic Text}},
  author = {Almohaimeed, Saleh and Almohaimeed, Saad and Jari, Mousa and Alobaid, Khaled A. and Alotaibi, Fahad},
  year = {2025},
  doi = {10.48550/arxiv.2511.16690},
  eprint = {2511.16690},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.16690}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asharq Al-Awsat. Asharq al-awsat: The leading pan-arab international newspaper, 2020. URL https://aawsat.com. Accessed 2025-11-08. Maged S. Al-Shaibani and Moataz Ahmed. The arabic ai fingerprint: Stylometric analysis and detection of large language models text. arXiv preprint arXiv:2505.23276 , 2025. Alibaba. Qwen-3: Next-generation multilingual foundation model. ↗ our card
  2. Alibaba Cloud Research Report , 2025. URL https://qwenlm.github.io/. Saad Almohaimeed. Thos: A benchmark dataset for targeted hate and offensive speech. In Proceedings of the Data-centric Machine Learning Research Workshop at ICML 2023 , 2023. arXiv preprint arXiv:2311.06446. Saad Almohaimeed. Transfer learning and lexicon-based approaches for implicit hate speech detection: A comparative study of human and gpt-4 annotation. In Proceedings of the 18th IEEE International Conference on Semantic Computing (ICSC-2024) , pages 142–147, 2024a. Saleh Almohaimeed. GAT-SQL: an advanced prompt engineering approach for effective text-to-sql interactions. In 2024 IEEE Congress on Evolutionary Computation (CEC) , pages xxx–xxx, 2024b. doi: 10.1109/CEC60901.2024.xxxxxx. Hamed Alshammari, Ahmed El-Sayed, and Khaled Elleithy. Ai-generated text detector for arabic language using encoder-based transformer architecture. Big Data and Cognitive Computing , 8(3):32, 2024. doi: 10.3390/bdcc8030032. Mohammed Altamimi. ANAD: arabic news article dataset. Data in Brief , 50:109460, 2023. doi: 10.1016/j.dib.2023.109460. Anthropic. Claude 4 sonnet: Advancements in scalable constitutional ai. ↗
  3. Anthropic Research Report , 2024. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, and . . . . ALLaM: large language models for arabic and english. arXiv preprint arXiv:2407.15390 , 2024. Rawad Baroud. Zerogpt: Ai content detection tool for gpt-generated text. ZeroGPT Research Blog . URL https://www.zerogpt.com. Chaka Chaka. Reviewing the performance of AI detection tools in differentiating between AI-generated and human-written texts: A literature and integrative hybrid review. Journal of Applied Learning & Teaching , 7(1):115–126, 2024. doi: 10.37074/ jalt.2024.7.1.14. Coins4Arab. Coins4arab – arabic cryptocurrency forum, 2020. URL https://coins4arab.com/vb/index.php. Accessed 2025-11-08. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020) , pages 8440–8451. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.747. Google DeepMind. Gemma 3: Open models for responsible ai innovation. ↗ our card
  4. Google DeepMind Technical Report , 2025. URL https://deepmind.google/technologies/ gemma/. Deepseek. Deepseek-v3.1: Advanced multilingual large language model. ↗
  5. DeepSeek AI Technical Report , 2025. URL https://www.deepseek.com/. Liam Dugan, Alyssa Hwang, Filip Trhlı́k, Josh Magnus Ludan, Andrew Zhu, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. Raid: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , pages 12463–12492, 2024. Omar Einea. SANAD: single-label arabic news articles dataset for automatic text categorization. Data in Brief , 25:104 076, 2019. doi: 10.1016/j.dib.2019.104076. Colleen Farrelly and Singh. Current topological and machine learning applications for bias detection in text. In 2023 6th International Conference on Signal Processing and Information Security (ICSPIS) , pages 190–195, 2023. doi: 10.1109/ ICSPIS60075.2023.10343824. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. GLTR: statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043 , 2019. Jon Gillham. Originality.ai: Advanced ai content detection and plagiarism analysis platform. Originality.AI Research Blog . URL https://originality.ai. Jonathan Gillham. Study finds almost 11 Originality.ai Blog , 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, and . . . . The LLaMA 3 Herd of Models. arXiv preprint arXiv:2407.21783 , 2024. IJSSA. The scientific journal of sport science & arts. URL https://ijssa.journals.ekb. eg/journal/about?lang=en. Accessed 2025-11-08. Isgen.ai. Isgen ai detector – detect ai-generated text (arabic & multilingual). URL https://isgen.ai/ar. Accessed 2025-11-09. Kevin. Smodin ai detector – accurate ai checker for chatgpt, gpt-5 & gemini. URL https://smodin.io/ai-content-detector. Accessed 2025-11-09. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. MAGE: machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , pages xxx–xxx, 2024. URL https://aclanthology.org/2024. acl-long.3.pdf. Dataset available at https://github.com/yafuly/MAGE. Dominik Macko, Robert Moro, Adaku Uchendu, Jason S. Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) , 2023. Mohammed Melhaj. EASC: essex arabic summaries corpus. Corpus release, 153 articles + 765 summaries, 2015. Available at SourceForge. Arthur Mensch. Mistral-seba 24: Efficient and open-weight large language model. ↗ our card
  6. Mistral AI Research Report , 2024. URL https://mistral.ai/news/mistral-7b/. Meta. The llama 3 herd of models. ↗
  7. Meta AI Research Report , 2024. Meta. LLaMA-4 Maverick-17B-128E-Instruct-FP8. https://huggingface.co/ meta-LLaMA/LLaMA-4-Maverick-17B-128E-Instruct-FP8, 2025. Mixture-of-Experts (MoE) model: 17B active params / 128 experts / FP8 quantization. ↗
  8. Released April 2025. Knowledge cutoff Aug 2024. :contentRefer-ence[oaicite:2]index=2. Moonshot. Kimi-K2-Instruct: A 1 Trillion-Parameter Mixture-of-Experts Language Model (32 B active) for Agentic Intelligence. https://huggingface.co/moonshotai/ Kimi-K2-Instruct, 2025. ↗
  9. Released July 2025; supports 128 K context window. :contentReference[oaicite:2]index=2. OpenAI. Openai platform – developer api & resources. URL https://platform.openai.com/. Accessed 2025-11-09. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 , 2023a. OpenAI. GPT-3.5-Turbo: Advancements in Instruction-Tuned Large Language Models. ↗ our card
  10. OpenAI Technical Report , 2023b. URL https://platform.openai.com/docs/ models/gpt-3-5-turbo. Accessed 2025-11-08. OpenAI. Gpt-4o: Omnimodal large language model. ↗
  11. OpenAI Technical Report , 2024. Shaina Raza, Muskan Garg, Deepak John Reji, Syed Raza Bashir, and Chen Ding. Nbias: A natural language processing framework for bias identification in text. Expert Systems With Applications , 237:121542, 2023. doi: 10.1016/j.eswa.2023. 121542. Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , page in press, 2020. ↗ our card
  12. Shoumik Saha and Soheil Feizi. Almost ai, almost human: The challenge of detecting ai-polished writing. Findings of the Association for Computational Linguistics: ACL 2025 , pages 25414–25431, 2025. Areg Mikael Sarvazyan, José Ángel González, Marc Franco-Salvador, Francisco Rangel, Berta Chulvi, and Paolo Rosso. Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domains. Procesamiento del Lenguaje Natural , 71:275–288, 2023. Nandika Sengupta, Sudip Kumar Sahu, Bo Jia, Sriram Katipomu, Hu Li, Felix Koto, Will Marshall, Gem Gosal, Chen Liu, Zhen Chen, and . . . . Jais and Jais-Chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 , 2023. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: masked and permuted pre-training for language understanding. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020) , pages 16857–16867. ↗ our card
  13. Curran Associates, Inc., 2020. doi: 10.5555/3495724.3497138. Zhenpeng Su. Hc3 plus: A semantic-invariant human chatgpt comparison corpus. Dataset release . . . , 2024. Supports wide variety of languages. TheDollzter. The dollzter travels – a travel and lifestyle blog featuring destinations, experiences, and guides around the world. URL https://www.thedollztertravels.com/. Accessed 2025-11-08. UN. United nations – official website: Global organization for peace, security, and cooperation. URL https://www.un.org. Accessed 2025-11-08. Hao Wang, Jianwei Li, and Zhengyu Li. Ai-generated text detection and classification based on bert deep learning algorithm. arXiv preprint arXiv:2405.16422 , 2024a. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohamed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1369–1407. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2024.eacl-long.83. URL https://aclanthology.org/2024.eacl-long. 83. Waqfeya. Waqfeya - arabic digital library for free downloadable books and manuscripts. URL https://waqfeya.com/. Accessed 2025-11-08. Wikipedia. Wikipedia: The free encyclopedia – a collaborative multilingual knowledge platform. URL https://ar.wikipedia.org. Accessed 2025-11-08. Han Xu, Jie Ren, Pengfei He, Shenglai Zeng, Yingqian Cui, Amy Liu, Hui Liu, and Jiliang Tang. On the Generalization of Training-based ChatGPT Detection Methods. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 7223–7243. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024. findings-emnlp.424. URL https://aclanthology.org/2024.findings-emnlp.424/. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almohaimeed-2025-ai-text-detectors-misclassification-slightly/