Elamur Research⌕
☰
home / papers / 2023 / Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

Summary

The paper investigates how well general-purpose chat LLMs handle Arabic, a language that makes up a small share of their training data. The authors ran GPT-3.5 and GPT-4, mostly zero-shot, on seven Arabic tasks: sentiment analysis, translation, transliteration (Arabizi to Egyptian Arabic), paraphrasing, part-of-speech tagging with UD tags, summarization, and diacritization. They compared the results with task-specific state-of-the-art systems.

Specialized Arabic models still win on most tasks. Summarization is the exception, where both chat models beat the prior best. GPT-4 is stronger than GPT-3.5 on five tasks, while GPT-3.5 leads on summarization and diacritization.

A deeper study of dialectal sentiment (AJGT) varies temperature, prompt wording and the number of demonstrations, and it inspects the raw model replies. The authors also release Taqyim, an open-source Python wrapper around OpenAI evals that makes such evaluations easier to run, resume and debug. The work gives a transparent, reproducible snapshot of where foundation models stand on Arabic and where gaps remain.

Key points

  • Seven tasks were tested on two models (GPT-3.5, GPT-4). GPT-4 is better on 5 of 7, and GPT-3.5 is better on summarization and diacritization.
  • Only in summarization did the chat models surpass existing SoTA systems. Diacritization results are additionally broken down across seven domains.
  • On the 360-sample AJGT test set, GPT-3.5 gave a usable label for only 342, 304, 317 and 312 samples at 0, 3, 5 and 10 shots.
    • Its failures included asking for an English version, explaining in Arabic, or refusing over offensive content.
    • Its zero-shot replies took more than 20 distinct forms.
  • GPT-4 almost always answered with the two allowed labels. The one exception was an invented "Neutral" label in the 3-shot setting.
  • Taqyim extends OpenAI evals with:
    • a Python interface;
    • Hugging Face datasets loading;
    • tiktoken-based context budgeting;
    • a resume_from_record option;
    • sequential requests so sample ids match dataset rows.

Interesting ideas

  • I1. For POS tagging, the model receives an already tokenized sentence and is told not to merge, split or alter tokens, so its output can be aligned row by row with the gold data. — For Mova (learn): LLM teacher tiers could get our own en tokens fixed in the prompt, letting their tags or arcs be compared directly with witness parsers when building silver data.
  • I2. Instead of mapping every reply onto the label set, the authors build a confusion matrix over the raw responses, which exposes off-template answers, refusals and invented classes. — For Mova (learn): when grading LLM teachers or our own readers, counting the unexpected outputs separately would act as a cheap check for format drift and hidden failure modes.
  • I3. Requests are sent in order and runs can resume from the last saved record, which keeps results traceable to specific inputs and avoids paying twice after a crash. — For Mova (learn): long teacher-labelling jobs could log by stable sentence id and resume, making silver-data provenance auditable and errors easy to locate.
  • I4. Each prompt is tried in several phrasings and with varying temperature and number of demonstrations, so a score reflects robustness rather than one lucky prompt. — For Mova (learn): reporting teacher agreement across several prompt variants would show which silver labels are stable and which depend on wording.
  • I5. Diacritization scores are reported per domain rather than as one average, which reveals where performance actually breaks down. — For Mova (tree): evaluating parsers and dialect conversion per genre (news, fables, school texts) would show which text types need more treebank data or repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

16 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.16322 · arXiv 2306.16322

@misc{alyafeai2023taqyim,
  title = {{Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models}},
  author = {Alyafeai, Zaid and Alshaibani, Maged S. and AlKhamissi, Badr and Luqman, Hamzah and Alareqi, Ebrahim and Fadel, Ali},
  year = {2023},
  doi = {10.48550/arxiv.2306.16322},
  eprint = {2306.16322},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.16322}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  2. Cascella, M., Montomoli, J., Bellini, V., and Elena, B. (2023). Evaluating the feasibility of chatgpt in healthcare: An analysis of multiple clinical and research scenarios. Journal of Medical Systems, 47. find
  3. Abdelali, A., Mubarak, H., Chowdhury, S. A., Hasanain, M., Mousi, B., Boughorbel, S., Kheir, Y. E., Izham, D., Dalvi, F., Hawasly, M., et al. (2023). Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982. ↗
  4. Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., and Song, Y. (2023). Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. arXiv preprint arXiv:2304.14827. ↗ our card
  5. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020). Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗ our card
  6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. ↗
  7. Alghamdi, A., Duan, X., Jiang, W., Wang, Z., Wu, Y., Xia, Q., Wang, Z., Zheng, Y., Rezagholizadeh, M., Huai, B., et al. (2023). Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint arXiv:2306.06800. ↗
  8. Alian, M., Awajan, A., Al-Hasan, A., and Akuzhia, R. (2021). Building arabic paraphrasing benchmark based on transformation rules. pages 1–17. find
  9. AlKhamissi, B., ElNokrashy, M., and Gabr, M. (2020). Deep diacritization: Efficient hierarchical recurrence for improved Arabic diacritization. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 38–48, Barcelona, Spain (Online). Association for Computational Linguistics. find
  10. Alomari, K. M., ElSherif, H. M., and Shaalan, K. (2017). Arabic tweets sentimental analysis using machine learning. In Advances in Artificial Intelligence: From Theory to Practice: 30th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2017, Arras, France, June 27-30, 2017, Proceedings, Part I 30, pages 602–610. Springer. find
  11. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023). A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. ↗ our card
  12. Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., Strassel, S., Habash, N., Eskander, R., and Rambow, O. (2014). Transliteration of Arabizi into Arabic orthography: Developing a parallel annotated Arabizi-Arabic script SMS/chat corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 93–103, Doha, Qatar. Association for Computational Linguistics. find
  13. Bommasani et al., R. (2021). On the opportunities and risks of foundation models. ArXiv, abs/2108.07258. ↗ our card
  14. Darwish, K., Mubarak, H., and Abdelali, A. (2017). Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17, Valencia, Spain. Association for Computational Linguistics. find our card
  15. Davis, E. (2023). Mathematics, word problems, common sense, and artificial intelligence. arXiv preprint arXiv:2301.09723. ↗
  16. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805. ↗
  17. El-Haj, M., Kruschwitz, U., and Fox, C. (2010). Using mechanical turk to create a corpus of arabic summaries. find
  18. Elmadany, A., Abdul-Mageed, M., et al. (2022a). Arat5: Text-to-text transformers for arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647. find our card
  19. Elmadany, A., Nagoudi, E. M. B., and Abdul-Mageed, M. (2022b). Orca: A challenging benchmark for arabic language understanding. ArXiv, abs/2212.10758. ↗
  20. Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. (2023). Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867. ↗ our card
  21. Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022). Revisiting pre-trained language models and their evaluation for arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151. find
  22. Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., and Chartash, D. (2022). How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment. medRxiv, pages 2022–12. find
  23. Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., and Wu, Y. (2023). How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. ↗ our card
  24. Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., and Awadalla, H. H. (2023). How good are gpt models at machine translation? a comprehensive evaluation. Minneapolis, Minnesota. Association for Computational Linguistics. find
  25. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022). Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. ↗ our card
  26. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  27. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. ↗
  28. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. (2023). Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. ↗ our card
  29. Jiao, W., Wang, W., Huang, J., Wang, X., and Tu, Z. (2023). Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745. ↗ our card
  30. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  31. Kondratyuk, D. and Straka, M. (2019). 75 languages, 1 model: Parsing universal dependencies universally. arXiv preprint arXiv:1904.02099. ↗ our card
  32. Ladhak, F., Durmus, E., Cardie, C., and McKeown, K. (2020). Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093. ↗ our card
  33. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗ our card
  34. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., and Ge, B. (2023). Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. find
  35. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692. ↗
  36. Lu, H., Huang, H., Ma, S., Zhang, D., Lam, W., and Wei, F. (2022). Trip: Triangular document-level pre-training for multilingual language models. arXiv preprint arXiv:2212.07752. ↗
  37. Mubarak, H., Abdelali, A., Sajjad, H., Samih, Y., and Darwish, K. (2019). Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395, Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. ↗ our card
  38. Shazal, A., Usman, A., and Habash, N. (2020). A unified model for arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the fifth arabic natural language processing workshop, pages 167–177. find
  39. Smrz, O., Bielickỳ, V., Kourilová, I., Krácmar, J., Hajic, J., and Zemánek, P. (2008). Prague arabic dependency treebank: A word on the million words. In Proceedings of the workshop on Arabic and local languages (LREC 2008), pages 16–23. find
  40. Tawkat Islam Khondaker, M., Waheed, A., Moatez Billah Nagoudi, E., and Abdul-Mageed, M. (2023). Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. arXiv e-prints, pages arXiv–2305. find
  41. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  42. Wang, T., Roberts, A., Hesslow, D., Le Scao, T., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. (2022). What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964–22984. PMLR. find
  43. Zeman et al., D. (2020). Universal dependencies 2.7. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  44. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223. ↗ our card
  45. Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016). The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534. find our card
  46. Zitouni, I. and Sarikaya, R. (2009). Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23:257–276. find
  47. Šlapeta, J. (2023). Are chatgpt and other pretrained language models good parasitologists? Trends in Parasitology, 39(5):314–316. *Figure 8: GPT-3.5 responses to AJGT dataset confusion matrix with 3 few-shot examples. Numbers are removed as most of them, except the highlighted areas, are zeros. Also, responses are removed for clarity purposes* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alyafeai-2023-taqyim-evaluating-arabic-nlp-tasks/