Elamur Research⌕
☰
home / papers / 2026 / Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models

Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models

Summary

This study asks whether current large language models can reproduce human affective expression closely enough to pass as human. It covers emotion in English and personality in Arabic, an under-resourced language. Six LLMs (Jais, Mistral, LLaMA, GPT-4o, Gemini, DeepSeek) were prompted to produce texts carrying target emotions or personality traits. Paraphrased versions of these outputs were also created.

Two experiments followed. First, classifiers were trained to separate human from machine authorship. Second, emotion and personality classifiers were trained and tested across human, AI and mixed data sources. Psycholinguistic profiling with LIWC and readability indices added an interpretable comparison.

Detection was very accurate (F1 above 0.95) but weakened considerably on paraphrased texts, which suggests the detectors depend on shallow stylistic cues. Classifiers transferred poorly between human and synthetic texts in both directions, so models seem to encode affect differently from people. Adding synthetic data helped only in the Arabic personality task. The work matters for AI-text detection, affective computing and low-resource data strategies.

Key points

  • Six models were compared: Jais, Mistral, LLaMA, GPT-4o, Gemini and DeepSeek, covering both multilingual and region-specific (Arabic-centric) systems.
  • Authorship detection:
    • Human-vs-AI detection reached F1 > 0.95.
    • Removing punctuation hurt it only moderately.
    • Paraphrasing caused a much sharper drop, especially on AI-authored samples.
  • Cross-source generalization: emotion and personality classifiers trained on human text did poorly on AI text, and the reverse also held. Mixing both sources gave modest gains for Arabic personality but no consistent gain for English emotion.
  • Model-level differences:
    • In English, GPT-4o and Gemini were more human-aligned on some emotions (e.g., disgust), whereas LLaMA performed poorly on the majority of emotion categories.
    • For the Arabic data, Jais and Mistral showed comparatively stronger alignment with personality cues.
  • Linguistic profiles (English):
    • LIWC was applied per sample and readability per corpus.
    • Human texts were more complex, with higher Gunning Fog and Dale-Chall scores, and differed in authenticity and emotional tone.
    • LLM texts were more readable and leaned analytic rather than narrative.
    • The Arabic dataset was small, and the data is available only on request.

Interesting ideas

  • I1. Using paraphrased copies as a stress test shows whether a classifier relies on real signal or on surface form; a big drop exposes shortcut learning. — For Mova (learn): Add paraphrase-based negative controls to evaluation, checking that frames and answers stay stable when a sentence is reworded with the same meaning.
  • I2. Training on one source and testing on the other (human→AI and AI→human) cleanly measures the distribution gap between synthetic and natural text. — For Mova (learn): Score models trained on LLM-teacher silver data separately on human-written gold and on synthetic text, so teacher artifacts cannot inflate the metrics.
  • I3. Synthetic augmentation helped the low-resource setting but not the well-resourced one, so the benefit of generated data depends on context and should be tested rather than assumed. — For Mova (learn): Apply LLM-generated training material selectively to rare constructions or sparse verb classes, and measure gains against a baseline without augmentation.
  • I4. Ablating a single surface feature (here punctuation) measures how much a model leans on formatting cues. — For Mova (tree): Run the perceptron parser and gparse on punctuation-stripped input to quantify how much attachment decisions depend on commas and periods.
  • I5. Asking generators to choose diverse topics reduces the risk that a target label becomes confounded with subject matter. — For Mova (answer): When generating word problems or fable-QA test items, vary topics deliberately so readers cannot pass by exploiting topic-answer correlations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.23251 · arXiv 2603.23251

@misc{alsadhan2026is,
  title = {{Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models}},
  author = {Alsadhan, Nasser A},
  year = {2026},
  doi = {10.48550/arxiv.2603.23251},
  eprint = {2603.23251},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.23251}
}

References

48 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbasian, M., Azimi, I., Feli, M., Rahmani, A. M., & Jain, R. (2024). Empathy through multimodality in conversational interfaces. arXiv preprint arXiv:2405.04777. ↗
  2. Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (pp. 9–15). European Language Resources Association. find our card
  3. Alhayan, F., & Himdi, H. (2024). Ensemble learning approach for distinguishing human and computer-generated Arabic reviews. PeerJ Computer Science, 10, e2345. find
  4. Alsadhan, N. A. (2025). A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions. arXiv preprint arXiv:2502.09128. ↗ our card
  5. Alsadhan, N., & Skillicorn, D. (2020). Sentiment is an Attitude not a Feeling. In 2020 International Conference on Data Mining Workshops (ICDMW) (pp. 1-6). IEEE. find
  6. Alshammari, H., El-Sayed, A., & Elleithy, K. (2024). Ai-generated text detector for arabic language using encoder-based transformer architecture. Big Data and Cognitive Computing, 8(3), 32. find
  7. Alyammi, Sarah (2025). Empathy-driven Arabic Conversational Chatbot using Pre-trained Transformer Model (Master's thesis, King Saud University). find
  8. Chraibi, K., Chaker, I., Dhassi, Y., & Zahi, A. (2024). MSAPersonality: a modern standard Arabic dataset for personality recognition. International Journal of Electrical & Computer Engineering (2088-8708), 14(4). find
  9. Dale, E., & Chall, J. S. (1948). A formula for predicting readability: Instructions. Educational research bulletin, 37-54. find
  10. Dandash, M., & Asadpour, M. (2025). Personality analysis for social media users using Arabic language and its effect on sentiment analysis. Social Network Analysis and Mining, 15(1), 6. find our card
  11. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics. find
  12. Dubey, A. (2024). Capturing Style Through Large Language Models-An Authorship Perspective (Doctoral dissertation, Purdue University Graduate School). find
  13. El Bahri, N., Itahriouan, Z., Abtoy, A., & Jamil, M. O. (2024). Personality analysis of students’ writing in social media-based learning environments. IEEE Access. find
  14. Essam, B. A., & Abdo, M. S. (2021). How do Arab tweeters perceive the COVID-19 pandemic?. Journal of psycholinguistic research, 50(3), 507-521. find
  15. Ferrario, A., Sedlakova, J., & Trachsel, M. (2024). The role of humanization and robustness of large language models in conversational artificial intelligence for individuals with depression: a critical analysis. JMIR Mental Health, 11, e56569. find
  16. Fitria, T. N. (2021). QuillBot as an online tool: Students’ alternative in paraphrasing and rewriting of English writing. Englisia: Journal of Language, Education, and Humanities, 9(1), 183-196. find
  17. Flesch, R. (1948). A new readability yardstick. Journal of applied psychology, 32(3), 221. Gallifant, J., Fiske, A., Levites Strekalova, Y. A., Osorio-Valencia, J. S., Parke, R., Mwavu, R., ... find
  18. & Pierce, R. (2024). Peer review of GPT-4 technical report and systems card. PLOS digital health, 3(1), e0000417. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... & Vasic, P. (2024). The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783. ↗ our card
  19. Gunning, R. (1952). The technique of clear writing. (No Title). find
  20. Hadi, M.U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M.B., Akhtar, N., Wu, J. and Mirjalili, S. (2023) ‘Large language models: a comprehensive survey of its applications, challenges, limitations, and prospects’, Authorea Preprints, Vol. 1, No. 3, pp.1–26. find
  21. Huang, B., Chen, C., & Shu, K. (2024). Can large language models identify authorship?. arXiv preprint arXiv:2403.08213. ↗
  22. Huang, B., Chen, C., & Shu, K. (2025). Authorship attribution in the era of llms: Problems, methodologies, and challenges. ACM SIGKDD Explorations Newsletter, 26(2), 21-43. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., las Casas, D. de, … Sayed, W. E. (2023). Mistral 7B. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2310.06825 ↗
  23. Klinkert, L. J., Buongiorno, S., & Clark, C. (2024). Evaluating the efficacy of LLMs to emulate realistic human personalities. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Vol. 20, No. 1, pp. 65-75). find
  24. Krook, J., Schneiders, E., Seabrooke, T., Leesakul, N., & Clos, J. (2024). Large Language Models (LLMs) for Legal Advice: A Scoping Review. Available at SSRN 4976189. find
  25. Lang, G., Triantoro, T., & Sharp, J. H. (2024). Large Language Models as AI-Powered Educational Assistants: Comparing GPT-4 and Gemini for Writing Teaching Cases. Journal of Information Systems Education, 35(3), 390-407. find
  26. Lee, S., Lim, S., Han, S., Oh, G., Chae, H., Chung, J., ... & Yu, Y. (2024). Do llms have distinct and consistent personality? trait: Personality testset designed for llms with psychometrics. arXiv preprint arXiv:2406.14703. ↗
  27. Lee, Y. K., Lee, I., Shin, M., Bae, S., & Hahn, S. (2024). Enhancing Empathic Reasoning of Large Language Models Based on Psychotherapy Models for AI-assisted Social Support. Korean Journal of Cognitive Science, 35(1). find
  28. Li, Z., Zhu, D., Ma, Q., Xiong, W., & Li, S. (2024). EERPD: Leveraging Emotion and Emotion Regulation for Improving Personality Detection. arXiv preprint arXiv:2406.16079. ↗
  29. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., ... & Piao, Y. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. ↗
  30. Mikros, G. (2025). Beyond the surface: stylometric analysis of GPT-4o’s capacity for literary style imitation. Digital Scholarship in the Humanities, 40(2), 587-600. find
  31. Nfaoui, E. H., & Elfaik, H. (2024). Evaluating Arabic Emotion Recognition Task Using ChatGPT Models: A Comparative Analysis between Emotional Stimuli Prompt, Fine-Tuning, and In-Context Learning. Journal of Theoretical and Applied Electronic Commerce Research, 19(2), 1118-1141. find
  32. Nidula Elgiriyewithana (2024). Dataset for Emotions: https://www.kaggle.com/datasets/nelgiriyewithana/emotions ↗
  33. Olteanu, A., Barocas, S., Blodgett, S. L., Egede, L., DeVrio, A., & Cheng, M. (2025). Ai automatons: Ai systems intended to imitate humans. arXiv preprint arXiv:2503.02250. ↗
  34. Opara, C. (2024). StyloAI: Distinguishing AI-generated content with stylometric analysis. In International conference on artificial intelligence in education (pp. 105-114). Cham: Springer Nature Switzerland. find
  35. Paech, S. J. (2023). Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281. ↗
  36. Pearson, J. (2024). Integration of Large Language Models in Marketing and Business Processes. find
  37. Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, 71(2001), 2001. find
  38. Ratican, J., & Hutson, J. (2024). Advancing sentiment analysis through emotionally-agnostic text mining in large language models (LLMS). Journal of Biosensors and Bioelectronics Research. find
  39. Roccas, S., Sagiv, L., Schwartz, S. H., & Knafo, A. (2002). The big five personality factors and personal values. Personality and social psychology bulletin, 28(6), 789-801. find
  40. Roy, B. N., Maity, S., Singh, K. R., Chowdhury, P., Das, A., & Saha, D. (2024). ECGText: Crafting Emotional Texts through GenAI. In 2024 International Conference on Intelligent Computing and Emerging Communication Technologies (ICEC) (pp. 1-6). IEEE. find
  41. Sengupta, N., Sahu, S. K., Jia, B., Katipomu, S., Li, H., Koto, F., ... & Xing, E. (2023). Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  42. Senter, R. J., & Smith, E. A. (1967). Automated readability index (No. AMRLTR66220). find
  43. Shreejit Cheela (2023). dataset for Text Emotion Recognition: https://www.kaggle.com/datasets/shreejitcheela/text-emotion-recognition ↗
  44. Taşkıran Tepe, H., & Aslantürk, H. (2025). Comparative Efficacy of AI LLMs in Clinical Social Work: ChatGPT-4, Gemini, Copilot. Research on Social Work Practice, 10497315241313071. find
  45. Team, G., Anil, R., Borgeaud, S., Alayrac, J. B., Yu, J., Soricut, R., ... & Blanco, L. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  46. Toygar Tanyel (2022). dataset for natural language processing: https://www.kaggle.com/datasets/toygarr/datasets-for-natural-language-processing ↗
  47. Wallace, B. C., Kertz, L., & Charniak, E. (2014). Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 512-516). find
  48. Zamir, M. T., Ayub, M. A., Gul, A., Ahmad, N., & Ahmad, K. (2024). Stylometry analysis of multi-authored documents for authorship and author style change detection. arXiv preprint arXiv:2401.06752. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alsadhan-2026-ai-catching-up-human-expression/