Elamur Research⌕
☰
home / papers / 2025 / AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data

AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data

Summary

LLMs are tested heavily on English tables, but no benchmark checked how they read and reason over Arabic tabular data. AraTable fills this gap. It gathers Arabic tables from three kinds of sources: Wikipedia, real-world open and government data, and tables produced by LLMs. Each table comes with questions of three types: direct lookup QA, fact verification and multi-step reasoning. LLMs wrote the first drafts of the questions and answers, and native Arabic-speaking experts then filtered and checked them. Four open models were tested zero-shot: Llama 3.3 70B, Mistral Large, DeepSeek-V3 and Jais 70B. DeepSeek-V3 scored best, Llama and Mistral came close behind, and Jais trailed far back. Every model found lookup questions easier than reasoning questions, which often stayed under 60% accuracy. The authors also built an automatic judge for free-text answers called assisted self-deliberation (ASD). Its scores almost match those of multi-round human raters, so evaluation on the benchmark can be cheap and repeatable.

Key points

  • The benchmark has three task types (direct QA, fact verification, reasoning) over three table sources (Wikipedia, real-world, LLM-generated). All data is public and comes from openly licensed sources.
  • Human-judged accuracy of DeepSeek-V3 was 0.79 on Wikipedia, 0.75 on real-world and 0.75 on LLM-generated tables. Jais 70B reached only 0.49, 0.31 and 0.45 on the same sets.
  • ASD was tested with Qwen and 4O as judges. After deliberation, Qwen matched the human accuracy exactly for every evaluated model on the Wikipedia and real-world sets, with signed gaps of +0.00.
  • Without deliberation, 4O underrated some systems by a wide margin, for example −0.15 and −0.16 for the Jais full-answer variant. Deliberation shrank but did not erase these gaps.
  • Limitations: only open Arabic-capable models were tested, only in zero-shot mode, and tables had about 40 rows each.

Interesting ideas

  • I1. A judge model first sees a second model's alternative verdict, then reconsiders, and the main judge still makes the final call; the second opinion pushes the judge into a more critical review instead of acting as a vote. — For Mova (learn): an LLM teacher ruling on disputed silver parses could be shown one other witness's rationale before deciding, which may cut out careless teacher errors.
  • I2. The automatic judge is checked against humans with a signed per-system gap, not one overall agreement number, which exposes judges that unfairly punish one particular system. — For Mova (learn): when an LLM grader scores Mova against baselines, report signed per-system gaps from a human-graded sample to catch graders biased against Mova's output style.
  • I3. Questions are split into lookup, verification and reasoning, so a high overall score cannot hide weak inference. — For Mova (answer): tag school-test and fable-QA items by whether the answer is read off the world directly or must be inferred, and track the two scores separately.
  • I4. The same tasks run over tables from several sources (encyclopedic, administrative, synthetic) to test whether skill carries over between domains. — For Mova (world): when testing holdings, measures and states, mix hand-written, real-document and synthetic world states to see if the extraction is robust across sources.
  • I5. An LLM drafts the question–answer pairs and native-speaker experts filter and verify them, a cheap way to build a benchmark that still keeps quality high. — For Mova (answer): have an LLM draft candidate questions about fables and word problems, then keep only human-checked items as a gold test set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.18442 · arXiv 2507.18442

@misc{alshaikh2025aratable,
  title = {{AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data}},
  author = {Alshaikh, Rana and Alghanmi, Israa and Jeawak, Shelan},
  year = {2025},
  doi = {10.48550/arxiv.2507.18442},
  eprint = {2507.18442},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.18442}
}

References

61 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Li, X., Feng, X., Hu, S., Wu, M., Zhang, D., Zhang, J., Huang, K.: Dtllm-vlt: Diverse text generation for visual language tracking based on llm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7283–7292 (2024) find
  2. Liu, X., Sun, T., Xu, T., Wu, F., Wang, C., Wang, X., Gao, J.: Shield: Evaluation and defense strategies for copyright compliance in llm text generation. arXiv preprint arXiv:2406.12975 (2024) ↗
  3. Koshkin, R., Sudoh, K., Nakamura, S.: Transllama: Llm-based simultaneous translation system. arXiv preprint arXiv:2402.04636 (2024) ↗
  4. Wang, L., Du, Z., Jiao, W., Lyu, C., Pang, J., Cui, L., Song, K., Wong, D., Shi, S., Tu, Z.: Benchmarking and improving long-text translation with large language models. In: Findings of the Association for Computational Linguistics ACL 2024, pp. 7175–7187 (2024) find
  5. Allemang, D., Sequeda, J.: Increasing the llm accuracy for question answering: Ontologies to the rescue! arXiv preprint arXiv:2405.11706 (2024) ↗
  6. Bui, T., Tran, O., Nguyen, P., Ho, B., Nguyen, L., Bui, T., Quan, T.: Cross-data knowledge graph construction for llm-enabled educational question-answering system: a case study at hcmut. In: Proceedings of the 1st ACM Workshop on AI-Powered Q&A Systems for Multimedia, pp. 36–43 (2024) find
  7. O’Brien Quinn, H., Sedky, M., Francis, J., Streeton, M.: Literature review of explainable tabular data analysis. Electronics 13 (19), 3806 (2024) find
  8. Nguyen, G., Brugere, I., Sharma, S., Kariyappa, S., Nguyen, A.T., Lecue, F.: Interpretable llm-based table question answering. arXiv abs/2412.12386 (2024) ↗
  9. Zhang, Z., Lin, X.V., Agrawal, R., Riedewald, M., Zhang, J.: Normtab: Improving symbolic reasoning in llms through tabular data normalization. In: Findings of the Association for Computational Linguistics: EMNLP 2024 (2024). https://api.semanticscholar.org/CorpusID:272645535 ↗
  10. Wang, Y., Zhang, S., Lu, Y., Wang, J., Wang, S., Zhang, X., Li, H.: Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios. arXiv abs/2403.19318 (2025) ↗
  11. Wu, X., Liang, D., Li, Z.: Tablebench: A comprehensive and complex benchmark for table question answering. arXiv abs/2408.09174 (2024) ↗
  12. Grijalba, J.O., Ureña-López, L.A., Camacho-Collados, J., Cámara, E.M.: Question answering over tabular data with databench: A large-scale empirical evaluation of llms. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 13471–13488 (2024). https://api.semanticscholar.org/CorpusID:266791020 ↗
  13. Xing, J., He, Y., Zhou, M., Dong, H., Han, S., Chen, L., Zhang, D., Chaudhuri, S., Jagadish, H.: Mmtu: A massive multi-task table understanding and reasoning benchmark. arXiv preprint arXiv:2506.05587 (2025) ↗
  14. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) ↗
  15. AI, M.: Mistral-Large (Version 1.0) [Large language model]. Hugging Face, https://huggingface.co/mistralai/Mistral-Large (2024) ↗
  16. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024) ↗
  17. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Marshall, W., Gosal, G., Liu, C., Chen, Z., et al.: Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023) ↗ our card
  18. Rhel, H., Roussinov, D.: Large language models and arabic content: A review. arXiv preprint arXiv:2505.08004 (2025) ↗ our card
  19. Mousi, S., al.: Aradice: Benchmarks for dialectal and cultural capabilities in llms. ACL Anthology (2025) find
  20. Sajjad, H., al.: Arabench: A multi-task benchmark for arabic natural language processing. Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC) (2020) find
  21. Seelawi, H., Tuffaha, I., Gzawi, M., Farhan, W., Talafha, B., Badawi, R., Sober, Z., Al-Dweik, O., Freihat, A.A., Al-Natsheh, H.: ALUE: Arabic language understanding evaluation. In: Habash, N., Bouamor, H., Hajj, H., Magdy, W., Zaghouani, W., Bougares, F., Tomeh, N., Abu Farha, I., Touileb, S. (eds.) Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 173– 184. Association for Computational Linguistics, Kyiv, Ukraine (Virtual) (2021). https://aclanthology.org/2021.wanlp-1.18/ ↗
  22. Abdul-Mageed, M., Elfardy, H., Diab, M.: Arlue: An arabic representation learning benchmark for universal evaluation. arXiv preprint arXiv:2109.00845 (2021) ↗
  23. Hasanaath, A., Alansari, A., Ashraf, A., Salmane, C., Luqman, H., Ezzini, S.: Arareasoner: Evaluating reasoning-based llms for arabic nlp. arXiv preprint arXiv:2506.08768 (2025) ↗
  24. Abdallah, A., Kasem, M., Abdalla, M., Mahmoud, M., Elkasaby, M., Elbendary, Y., Jatowt, A.: Arabicaqa: A comprehensive dataset for arabic question answering. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2049–2059 (2024) find our card
  25. Abdaljalil, S., Kurban, H., Serpedin, E.: Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. arXiv preprint arXiv:2503.07833 (2025) ↗
  26. Alghamdi, E.A., Masoud, R., Alnuhait, D., Alomairi, A.Y., Ashraf, A., Zaytoon, M.: Aratrust: An evaluation of trustworthiness for llms in arabic. In: Proceedings of the 31st International Conference on Computational Linguistics, pp. 8664–8679 (2025) find
  27. Sui, Y., Zhou, M., Zhou, M., Han, S., Zhang, D.: Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 645–654 (2024) find
  28. Liu, T., Wang, F., Chen, M.: Rethinking tabular data understanding with large language models. In: Duh, K., Gomez, H., Bethard, S. (eds.) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 450–482. Association for Computational Linguistics, Mexico City, Mexico (2024). https://doi.org/10.18653/v1/2024.naacl-long.26 . https://aclanthology.org/2024.naacl-long.26/ ↗
  29. Ziqi, J., Lu, W.: Tab-cot: Zero-shot tabular chain of thought. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 10259–10277 (2023) find
  30. Wang, Z., Zhang, H., Li, C.-L., Eisenschlos, J.M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang, J., Lee, C.-Y., et al.: Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398 (2024) ↗
  31. Deng, N., Sun, Z., He, R., Sikka, A., Chen, Y., Ma, L., Zhang, Y., Mihalcea, R.: Tables as texts or images: Evaluating the table reasoning ability of llms and mllms. In: Findings of the Association for Computational Linguistics ACL 2024, pp. 407–426 (2024) find
  32. Chen, W.: Large language models are few(1)-shot table reasoners. In: Vlachos, A., Augenstein, I. (eds.) Findings of the Association for Computational Linguistics: EACL 2023, pp. 1120–1130. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/v1/2023.findings-eacl.83 . https://aclanthology.org/2023.findings-eacl.83/ ↗ our card
  33. Grijalba, J.O., Lopez, L.A.U., Martı́nez-Cámara, E., Camacho-Collados, J.: Question answering over tabular data with databench: A large-scale empirical evaluation of llms. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 13471–13488 (2024) find
  34. Bhandari, K.R., Xing, S., Dan, S., Gao, J.: On the robustness of language models for tabular question answering. arXiv preprint arXiv:2406.12719 (2024) ↗
  35. Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., Liu, Y.: Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 (2024) ↗ our card
  36. Lin, Y.-T., Chen, Y.-N.: Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711 (2023) ↗ our card
  37. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023) ↗
  38. Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., Liu, Z.: Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023) ↗ our card
  39. Chu, Z., Ai, Q., Tu, Y., Li, H., Liu, Y.: Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641 (2024) ↗
  40. Li, Q., Cui, L., Kong, L., Bi, W.: Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation. arXiv e-prints, 2310 (2023) find
  41. Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., Ma, X.: Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making. In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–23 (2025) find
  42. Lee, N., Hong, J., Thorne, J.: Evaluating the consistency of llm evaluators. In: Proceedings of the 31st International Conference on Computational Linguistics, pp. 10650–10659 (2025) find
  43. Panickssery, A., Bowman, S., Feng, S.: Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems 37 , 68772– 68802 (2024) find
  44. Zhang, X., Li, Y., Wang, J., Sun, B., Ma, W., Sun, P., Zhang, M.: Large language models as evaluators for recommendation explanations. In: Proceedings of the 18th ACM Conference on Recommender Systems, pp. 33–42 (2024) find
  45. Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470–1480 (2015) find
  46. Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., Chua, T.-S.: Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3277–3287 (2021) find our card
  47. Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., Wang, W.Y.: Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164 (2019) ↗ our card
  48. Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al. : Tablebench: A comprehensive and complex benchmark for table question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 25497–25506 (2025) find
  49. Kweon, S., Kwon, Y., Cho, S., Jo, Y., Choi, E.: Open-wikitable: Dataset for open domain question answering with complex reasoning over table. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 8285–8297 (2023) find
  50. Zhu, J., Wang, J., Yu, B., Wu, X., Li, J., Wang, L., Xu, N.: Tableeval: A real-world benchmark for complex, multilingual, and multi-structured table question answering. arXiv preprint arXiv:2506.03949 (2025) ↗
  51. Berkovitch, Y., Glickman, O., Somech, A., Wolfson, T.: Generating tables from the parametric knowledge of language models. arXiv preprint arXiv:2406.10922 (2024) ↗
  52. Elnagar, A., Khalifa, Y.S., Einea, A.: Hotel arabic-reviews dataset construction for sentiment analysis applications 740 , 35–52 (2018) https://doi.org/10.1007/978-3-319-67056-0 3 ↗
  53. Fang, X., Xu, W., Tan, F.A., Zhang, J., Hu, Z., Qi, Y., Nickleach, S., Socolinsky, D., Sengamedu, S., Faloutsos, C.: Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey. arXiv preprint arXiv:2402.17944 (2024) ↗
  54. Nguyen, N.-H., Sim, T., Dao, H., Joty, S., Kawaguchi, K., Chen, N., Kan, M.-Y., et al. : Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 299–330 (2025) find
  55. Tam, Z.R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., Chen, Y.-N.: Let me speak freely? a study on the impact of format restrictions on large language model performance. In: Dernoncourt, F., Preoţiuc-Pietro, D., Shimorina, A. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1218–1236. Association for Computational Linguistics, Miami, Florida, US (2024). https://doi.org/10.18653/v1/2024. emnlp-industry.91 . https://aclanthology.org/2024.emnlp-industry.91/ ↗
  56. Xia, C., Xing, C., Du, J., Yang, X., Feng, Y., Xu, R., Yin, W., Xiong, C.: Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667 (2024) ↗
  57. Lakkaraju, H., Slack, D., Chen, Y., Tan, C., Singh, S.: Rethinking explainability as a dialogue: A practitioner’s perspective. arXiv preprint arXiv:2202.01875 (2022) ↗
  58. Qwen: Qwen: A Large Language Model by Tongyi Lab. https://qwenlm.github.io/. Accessed: 2023-10-01 (2023) ↗
  59. OpenAI: GPT-4o (Version gpt-4o-latest). Large language model (2024). https://openai.com/index/hello-gpt-4o ↗
  60. Liu, T., Wang, X., Huang, W., Xu, W., Zeng, Y., Jiang, L., Yang, H., Li, J.: Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409.14051 (2024) ↗
  61. Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate. In: Forty-first International Conference on Machine Learning (2023) find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshaikh-2025-aratable-benchmarking-llms-reasoning-understanding/