Elamur Research⌕
☰
home / papers / 2024 / The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic

The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic

Paper · 2024 · Shahad Al-Khalifa and Hend Al-Khalifa

Summary

Arabic lacks native, high-quality benchmarks for checking how well large language models reason about mathematics and understand language. Most earlier resources were translated from English. The authors address this by building a benchmark from the Qiyas General Aptitude Test (GAT), the Saudi university-admission exam. Domain experts experienced in writing and grading that exam authored the items. The resulting set has 2,409 four-option multiple-choice questions. The quantitative section spans math, geometry, algebra and statistics, while the verbal section includes five item types: verbal analogy, anomalous word, contextual error, sentence completion, and comprehension of reading passages. Items that depend on figures were left out.

ChatGPT-3.5-turbo and ChatGPT-4 were queried through the OpenAI API in zero-, one- and 3-shot settings. The prompts reused the official Arabic exam instructions plus an answer-only directive, and accuracy was the metric. ChatGPT-4 averaged about 64% and ChatGPT-3.5-turbo about 49%. Extra in-context examples usually helped.

Questions both models missed were then given to Gemini-pro, which recovered a share of them. Verbal analogy remained hard for every model. The benchmark gives Arabic LLM work a demanding, expert-validated yardstick.

Key points

  • The dataset has 2,409 questions in total, split into 1,014 quantitative and 1,395 verbal items:
    • Quantitative: Math 370, Geometry 142, Algebra 374, Statistics 128.
    • Verbal: Reading Comprehension 199, Sentence Completion 300, Contextual Error 298, Verbal Analogy 298, Anomalous Word 300.
  • Overall accuracy was roughly 64% for ChatGPT-4 and 49% for ChatGPT-3.5-turbo.
  • ChatGPT-4 did best on math and statistics with zero-shot prompts. It peaked at 81% on geometry and 63% on algebra with 3-shot prompts.
  • Error rates differed by model and task:
    • Contextual Error was ChatGPT-4's weakest verbal type, with 132/298 wrong; near-synonymous options were a likely cause.
    • Verbal Analogy was ChatGPT-3.5-turbo's weakest, with 197/298 wrong.
    • On the quantitative side, probability and combinatorics caused trouble.
  • Gemini-pro was re-run on the failed items:
    • It solved 16–56% of them depending on type, doing best on Reading Comprehension and worst on Verbal Analogy.
    • It also followed the answer-only instruction more reliably than either ChatGPT model. ChatGPT-3.5 often ignored it.

Interesting ideas

  • I1. Prompts were taken word-for-word from the official exam instructions, not written by the evaluators, so the test mirrors what human examinees see and avoids prompt-engineering bias.
  • I2. Items that need a picture or chart were deliberately excluded, keeping the benchmark a clean measure of text-only ability.
  • I3. The "contextual error" task asks which single word clashes with the overall meaning of an otherwise grammatical sentence, which is a crisp test of semantic coherence.
  • I4. Items one model failed were re-run on a different model to separate model-specific errors from task-wide difficulty; verbal analogy stayed hard for everyone.
  • I5. The one-shot and 3-shot examples were held fixed across all questions, and compliance with an answer-only format was treated as an observable behaviour in its own right.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course tests could be fed to the readers using the exact original task wording, so scores compare fairly with human pupils and are not inflated by hand-tuned phrasing.
  • I2 (answer): Maths word-problem sets could be filtered for items whose solution depends on missing diagrams, so a failure points to a genuine parsing or world-building gap, not an absent input.
  • I3 (world): One could build English items of this kind and check whether the world of the text flags the entity or state that contradicts the rest, giving an explainable coherence probe.
  • I4 (learn): When the parser or reader fails, re-checking those cases with other witness parsers or an LLM teacher tier could show whether the error belongs to Mova or reflects inherently ambiguous data, before any repair rules are induced.
  • I5 (learn): Evaluations of LLM teachers could keep demonstrations constant and log instruction-following separately from correctness, so noisy teacher output is caught before it enters silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Saud UniversityRiyadh · Saudi Arabia37 works · 527 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Shahad Al-Khalifa and Hend Al-Khalifa. 2024. The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic. Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alkhalifa2024qiyas,
  title = {{The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic}},
  author = {Shahad Al-Khalifa and Hend Al-Khalifa},
  booktitle = {Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.icnlsp-1.35/}
}

References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alkaoud, M. (2024). A bilingual benchmark for evaluating large language models. PeerJ Computer Science, 10, e1893. https://doi.org/10.7717/peerj-cs.1893 ↗
  2. Almansor, E. H., Al-Ani, A., & Hussain, F. K. (2020). Transferring Informal Text in Arabic as Low Resource Languages: State-of-the-Art and Future Research Directions. In L. Barolli, F. K. Hussain, & M. Ikeda (Eds.), Complex, Intelligent, and Software Intensive Systems (pp. 176–187). Springer International Publishing. https://doi.org/10.1007/978-3-030-22354-0_17 ↗
  3. Chow, J. C. L., Sanders, L., & Li, K. (2023). Impact of ChatGPT on medical chatbots as a disruptive technology. Frontiers in Artificial Intelligence, 6. https://doi.org/10.3389/frai.2023.1166014 ↗
  4. Education and Training Evaluation Commission. (n.d.). Education and Training Evaluation Commission. Retrieved April 1, 2024, from https://www.etec.gov.sa/en/qiyas ↗
  5. ETEC. (n.d.). بيردتلاو ميلعتلا ميوقت ةئيه —Media Center. Eucation and Training Evaluation Commission | ETEC. Retrieved April 4, 2024, from https://spapi.etec.gov.sa:2443/ar/MediaAsset s/aptbroch.pdf?csf=1&e=zc4F99 Feng, T. H., Denny, P., Wuensche, B., Luxton-Reilly, A., & Hooper, S. (2024). More Than Meets the AI: Evaluating the performance of GPT-4 on Computer Graphics assessment questions. Proceedings of the 26th Australasian Computing Education Conference, 182–191. https://doi.org/10.1145/3636243.3636263 ↗
  6. Fowler, T., Pullen, S., & Birkett, L. (2023). Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions. British Journal of Ophthalmology. https://doi.org/10.1136/bjo-2023-324091 ↗
  7. Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has Aced the Test of Understanding in College Economics: Now What? The American Economist, 68(2), 233–245. https://doi.org/10.1177/05694345231169654 Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., … Vinyals, O. (2024). Gemini: A Family of Highly Capable Multimodal Models (arXiv:2312.11805). arXiv. http://arxiv.org/abs/2312.11805 ↗
  8. Giannos, P., & Delardas, O. (2023). Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA Examinations. JMIR Medical Education, 9(1), e47737. https://doi.org/10.2196/47737 ↗
  9. Kaye, A. S. (2018). Arabic. In The World’s Major Languages (3rd ed.). Routledge. find
  10. Nguyen, X.-H., Nguyen, H.-A., Cao, L., & Hana, T. (2023). ChatGPT’s Understanding of History A Comparison to Vietnamese Students and its Potential in History Education. OSF. https://doi.org/10.35542/osf.io/8z9tj ↗
  11. OECD & The World Bank. (2022). How Learning Continued during the COVID-19 Pandemic: Global Lessons from Initiatives to Support Learners and Teachers (S. Vincent-Lancrin, C. Cobo Romaní, & F. Reimers, Eds.). OECD. https://doi.org/10.1787/bbeca162-en ↗
  12. OpenAI Platform. (n.d.). Retrieved April 4, 2024, from https://platform.openai.com Temsah, O., Khan, S. A., Chaiah, Y., Senjab, A., Alhasan, K., Jamal, A., Aljamaan, F., Malki, K. H., Halwani, R., Al-Tawfiq, J. A., Temsah, M.-H., Al-Eyadhy, A., Temsah, O., Khan, S. A., Chaiah, Y., Senjab, A. M., Alhasan, K., Jamal, A., Aljamaan, F., … Al-Eyadhy, A. (2023). Overview of Early ChatGPT’s Presence in Medical Literature: Insights From a Hybrid Literature Review by ChatGPT and Human Experts. Cureus, 15(4). https://doi.org/10.7759/cureus.37281 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalifa-2024-qiyas-benchmark-measuring-chatgpt-mathematical/