Elamur Research⌕
☰
home / papers / 2025 / Exploring AI Capabilities in Arabic Grammar: Comparative Analysis of ChatGPT and Gemini

Exploring AI Capabilities in Arabic Grammar: Comparative Analysis of ChatGPT and Gemini

Summary

The paper investigates how well two general-purpose chatbots, ChatGPT and Gemini, handle i'rāb (case and role analysis) of Marfû'ât al-Asmâ', the Arabic nouns that take the nominative. Learners find this topic hard, and an error in i'rāb can change how a sentence is understood. The authors used Fuad Ni'mah's textbook Mulakhash Qawâ'id al-Lugah al-'Arabiyyah to build 11 example sentences, one for each nominative category, from mubtada' and khabar through fā'il, nā'ib al-fā'il, na'at, 'atf, tawkid and badal. Both models received identical prompts. Raters scored every answer against the textbook on a 5-point Likert rubric, and the two sets of scores were compared with a Mann-Whitney test. A Cosine Similarity Index (CSI) was also computed on the answers. ChatGPT scored significantly higher (p = 0.019) and gave fuller, more accurate analyses. Gemini's answers were shorter and less precise. Even so, the CSI of 0.800 showed that the two models' outputs were textually very close. The authors conclude that both tools can support Arabic grammar teaching, but their output still needs manual correction.

Key points

  • Test set: 11 examples, selected purposively from Fuad Ni'mah's Mulakhash, one for each of 11 nominative categories: the mubtada' and its khabar; the fā'il and nā'ib al-fā'il; the khabar of inna, the isim of kāda and the isim ashbah; and na'at, 'atf, tawkid and badal.
  • Scoring used a 1–5 Likert rubric. A 5 requires agreement with the reference on grammatical position, mu'rab/mabni status, final letter and role. A 4 allows minor deviations such as a different final letter. A 3 means broadly right but with detail errors, such as wrong mu'rab/mabni status.
  • Statistics: a Mann-Whitney U test, chosen because the scores are ordinal and not normally distributed, gave a significant difference (0.019) in ChatGPT's favour.
  • The CSI is built from word-frequency vectors and was 0.800, indicating high similarity between the two models' outputs.
  • ChatGPT suits detailed, explanatory analysis, while Gemini suits quick, short answers. Both need human verification.

Interesting ideas

  • I1. The rubric breaks a single word's analysis into separate attributes (role, inflectional status, final marker) and gives partial credit for each, so a near-miss scores differently from a total miss. — For Mova (learn): Grading teacher-LLM or witness-parser outputs could use a similar ladder (head right / relation right / features right), giving finer signals for silver-data agreement than pass/fail.
  • I2. The probe set is small and category-balanced, built from textbook examples with exactly one item per grammatical function, which makes gaps visible per category. — For Mova (tree): A textbook-derived probe sentence per UD relation (e.g. appos for badal-like cases, conj for 'atf, nsubj:pass for nā'ib al-fā'il) would expose relation-specific weaknesses of the perceptron parser and gparse.
  • I3. Both models received exactly the same prompt, so any score difference reflects the models rather than the prompt wording. — For Mova (learn): When ranking LLM teacher tiers, freezing the prompt per task keeps tier comparisons fair and reproducible.
  • I4. The two models' outputs were very similar on surface cosine yet differed significantly in rated correctness, which shows that word-overlap similarity is a poor stand-in for structural accuracy. — For Mova (learn): This is a ready-made argument for a negative control in evaluation: bag-of-words similarity should be reported only alongside tree- or frame-level checks, never as a correctness metric.
  • I5. The authors judge that detailed answers are more useful for teaching but still require a human check before use. — For Mova (answer): For school-course tests, answers could carry the explainable grammatical derivation so that a teacher can verify each step quickly instead of trusting a bare verdict.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.15408/a.v11i2.42671

@inproceedings{maulidiya2025exploring,
  title = {{Exploring AI Capabilities in Arabic Grammar: Comparative Analysis of ChatGPT and Gemini}},
  author = {Maulidiya, Amanda Resti and Abdurrahman, Maman and Saleh, Nalahuddin},
  booktitle = {Arabiyat 11},
  year = {2025},
  doi = {10.15408/a.v11i2.42671},
  url = {https://doi.org/10.15408/a.v11i2.42671}
}

References

29 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali, M. A. (2016). Artificial intelligence and natural language processing: The Arabic corpora in online translation software. International Journal of Advanced and Applied Sciences, 3(9). find
  2. Aljanabi, M. (2024). Assessing the Arabic Parsing Capabilities of ChatGPT and Cloude: An Expert-Based Comparative Study. Mesopotamian Journal of Arabic Language Studies, 2024. find
  3. Anwar, M. R., & Ahyarudin, H. A. (2023). AI-Powered Arabic Language Education in the Era of Society 5.0. IAIC Transactions on Sustainable Digital Innovation (ITSDI), 5(1). find
  4. Berliani, D., Azhari, I., & Taufiqurrochman, R. (2024). Exploring The Potential Of Chatgpt As A Technology-Based Tool For Arabic Language Learning: Benefits And Risks. Kitaba, 2(2). find
  5. Elkhatat, A. M., Elsaid, K., & Almeer, S. (2023). Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity, 19(1). find
  6. Gomez-Cabello, C. A., Borna, S., Pressman, S. M., Haider, S. A., & Forte, A. J. (2024). Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini. Medicina, 60(6). find
  7. Khalatia, M., & Al-Romanyb, T. A. H. (2020). Artificial intelligence development and challenges (Arabic language as a model). Artificial Intelligence, 13(5). find
  8. Khidhir, A. M. (2024). An AI model for Parsing the Text of Holy Quran Sentences. Mesopotamian Journal of Quran Studies, 2024. find
  9. Kishan, A. U., & Reiter, R. E. (2021). Caught Between a Rock and a Hard Place. International Journal of Radiation Oncology, Biology, Physics, 109(4). find
  10. Lahitani, A. R., Permanasari, A. E., & Setiawan, N. A. (2016). Cosine similarity to determine similarity measure: Study case in online essay assessment. 2016 4th International Conference on Cyber and IT Service Management. find
  11. Mannaa, Z. M., Azmi, A. M., & Aboalsamh, H. A. (2022). Computer-assisted i ‘raab of Arabic sentences for teaching grammar to students. Journal of King Saud University-Computer and Information Sciences, 34(10). find
  12. Marsiah, M., Mubarak, M. R., & Audina, N. A. (2021). The students’ perspective towards youtube as the replacement of lecturer in nahwu learning. Jurnal Al Bayan: Jurnal Jurusan Pendidikan Bahasa Arab, 13(2). find
  13. Maulida, M., & Marsiah, M. (2023). Mapping Kesulitan Dalam Membaca dan Mengharokati Teks Bahasa Arab Bagi Mahasiswa Pendidikan Bahasa Arab. Journal Of Foreign Language Learning and Teaching, 3(2). find
  14. Milenović, Ž. (2011). Application Of Mann-Whitney U Test In Research Of Professional Training Of Primary School Teachers. Metodički Obzori/Methodological Horizons, 6(1). find
  15. Mualif, A. (2019). Metodologi pembelajaran ilmu nahwu dalam pendidikan bahasa Arab. AL-HIKMAH (Jurnal Pendidikan Dan Pendidikan Agama Islam), 1(1). find
  16. Muawanah, R., & Rifa’i, A. F. (2018). Analisis Kedudukan I’rab Kalimah Bahasa Arab Pada Kitab Al-Imrithi Menggunakan Algoritma Breadth First Search (BFS). JISKA (Jurnal Informatika Sunan Kalijaga), 3(1). find
  17. Musthafa, F. A. D. (2024). Penggunaan Artificial Intelligence (AI) dalam Pembelajaran: Fenomena Transformasi Otoritas Pengetahuan di Kalangan Mahasiswa. Journal of Contemporary Islamic Education, 4(1). find
  18. Nurmala, M., Sopian, A., Saleh, N., & Syihabuddin, S. (2022). Why is Masdar Important? An Investigating of Masdar and its Translation. Alsuniyat: Jurnal Penelitian Bahasa, Sastra, Dan Budaya Arab, 5(1). find
  19. Oti, E. U., Olusola, M. O., & Esemokumo, P. A. (2021). Statistical Analysis of the Median Test and the Mann-Whitney U Test. 7(9). find
  20. Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., & Forte, A. J. (2024). AI in Hand Surgery: Assessing Large Language Models in the Classification and Management of Hand Injuries. Journal of Clinical Medicine, 13(10). find
  21. Qadir, J. (2023). Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. find
  22. Rahmat, A. S., & Abdurrahman, M. (2017). Analysis of Nahwu Learning Progress in Indonesian Universities Based on the Character of Value Concerns. Tawarikh, 8(2). find
  23. Ramli, M. (2023). Mengeksplorasi tantangan etika dalam penggunaan Chat GPT sebagai alat bantu penulisan ilmiah: Pendekatan terhadap integritas akademik. TA’DIBAN: Journal of Islamic Education, 4(1). find
  24. Rane, N., Choudhary, S., & Rane, J. (2024). Gemini or ChatGPT? Capability, Performance, and Selection of Cutting-edge Generative Artificial Intelligence (AI) in Business Management. Capability, Performance, and Selection of Cutting-Edge Generative Artificial Intelligence (AI) in Business Management (February 19, 2024). find
  25. Siregar, F. H., Hasmayni, B., & Lubis, A. H. (2023). The analysis of chat gpt usage impact on learning motivation among scout students. International Journal of Research and Review, 10(7). find
  26. Suryono, M. N. R. N., Bhagaskara, R. E., Pratama, M. A., & Pratama, A. (2023). Analisis Pengaruh ChatGPT Terhadap Produktivitas Mahasiswa. 3(1). find
  27. Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1). find
  28. Wijonarko, P., Wagiman, W., Khana, R., Tundo, T., Salam, A., & James, B. (2023). Penerapan dan Kontribusi Kecerdasan Buatan ChatGPT Untuk Menafsir Teks Hukum (Studi Kasus Penafsiran Pasal 10, Pasal 13, Permenkes No. 889 Tahun 2011). Jurnal Kajian Teknik Elektro, 8(2). find
  29. Yatri, D., Anugerahwati, M., & Setyowati, L. (2023). Artificial Intelligence (Ai) In Language Learning (English And Arabic Class): Students’and Teachers’experience And Perceptions. Transformational Language, Literature, And Technology Overview In Learning, 3(1). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maulidiya-2025-exploring-ai-capabilities-arabic-grammar/