Elamur Research⌕
☰
home / papers / 2025 / CAMEL-Bench: A Comprehensive Arabic LMM Benchmark

CAMEL-Bench: A Comprehensive Arabic LMM Benchmark

Paper · 2025 · Sara Ghaboura, Ahmed Heakl, Omkar Thawakar et al.

Summary

Most benchmarks for large multimodal models (LMMs) are built around English. This leaves Arabic, with more than 400 million speakers, without a broad way to test models. CAMEL-Bench fills that gap. It is an Arabic evaluation suite covering eight domains and 38 sub-domains:

  • general multimodal reasoning
  • OCR and documents
  • charts and diagrams
  • video
  • culture-specific content
  • medical imaging
  • agriculture
  • remote sensing

The data has three origins: existing Arabic resources, English benchmarks machine-translated with GPT-4o, and newly collected web material. For the new material, GPT-4o wrote multiple-choice questions from curated context. A pool of roughly 41k questions was filtered down to 29,036 through two checks:

  • sampled manual audits for native Arabic data
  • model-based semantic comparison for translated data, with human review of anything rejected

Scoring uses exact match, edit distance or GPT-4o-judged fuzzy matching, depending on the task type. Even the strongest model, GPT-4o, reaches only about 62% overall. Remote sensing and Arabic OCR are the hardest areas, and open-source models lag well behind. The benchmark gives Arabic LMM development a public, diverse yardstick.

Key points

  • Scale: 8 domains, 38 sub-domains and 29,036 questions, filtered from over 41k candidates. The authors compare it with Exams-V, CVQA, Henna and KHATT, which are far narrower.
  • Audit of native Arabic data: a random 20% of each sub-category is checked by hand. If errors stay under 40%, the sub-category is accepted; otherwise every item in it is reviewed.
  • Check of translated data: Qwen7B, given 5 few-shot examples, compares each English QA pair with its Arabic version. Rejected pairs go to people, who route them into one of three outcomes:
    • full manual re-translation
    • model-assisted refinement
    • re-sending items that were never translated
  • Models tested: GPT-4o, GPT-4o-mini, Gemini-1.5-Pro/Flash, Pangea-7B, Qwen2-VL-2B, InternVL2-8B and LLaVa-NeXt-7B.
  • GPT-4o results: it leads in most domains, for example 80.86 on cultural and 80.75 on agro tasks, but scores only 22.85 on remote sensing.
  • Arabic script: ligatures and diacritics make handwritten and historical Arabic OCR (e.g., KHATT) especially difficult.
  • Coverage limit: the benchmark is mostly Modern Standard Arabic, and dialects are largely left out.

Interesting ideas

  • I1. Sampled audit with escalation: check a fixed random fraction of each data slice and review the whole slice only if the sampled error rate crosses a threshold, which spends human effort only where it is needed.
  • I2. Cheap screen, then triaged repair: a small model compares a translated item with its source, and failures are sorted into distinct repair routes (redo from scratch, light fix, never translated) rather than one generic "bad" bin.
  • I3. Metric matched to answer type: exact match is used for closed choices, edit distance for transcription, and a context-aware judge for open answers with synonyms, instead of forcing one score on all tasks.
  • I4. Constrained, grounded question generation: generated questions must carry no hints, use four distinct non-synonymous options, and be answerable from the given material alone, without outside knowledge.
  • I5. Guessing inflates scores: the authors note that multiple-choice and yes/no tasks look easier partly because of chance-level guessing, so raw accuracy needs a chance baseline beside it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver treebank batches agreed by witness parsers could be audited per genre or per source this way, with a whole batch sent back for re-parsing when its sample fails.
  • I2 (back): Round-trip output from the text → graph → text compressor could be screened against the source the same way, with failures labelled by repair type to guide what to fix in generation.
  • I3 (answer): School-test and fable-QA readers could be scored by question type: exact for multiple choice, numeric tolerance for maths word problems, and lenient matching for free-text answers.
  • I4 (answer): Auto-generated questions over tales could follow these rules, so a correct answer proves the reader built the text's world rather than leaned on priors.
  • I5 (learn): Evaluations could always report a random-choice or answer-shuffled negative control next to the real score, exposing gains that come from guessing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓Australian National UniversityCanberra · Australia23 works · 6,522 citations🎓Linköping UniversityLinköping · Sweden18 works · 2,319 citations🎓Aalto UniversityEspoo · Finland13 works · 648 citations🎓University of BradfordBradford · United Kingdom2 works · 278 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Husain Salem Abdulla Alharthi, Ines Riahi, Abduljalil Radman, Jorma Laaksonen, Fahad Shahbaz Khan, Salman Khan and Rao Muhammad Anwer. 2025. CAMEL-Bench: A Comprehensive Arabic LMM Benchmark. Findings of the Association for Computational Linguistics: NAACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-naacl.105

@inproceedings{ghaboura2025camel,
  title = {{CAMEL-Bench: A Comprehensive Arabic LMM Benchmark}},
  author = {Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Husain Salem Abdulla Alharthi, Ines Riahi, Abduljalil Radman, Jorma Laaksonen, Fahad Shahbaz Khan, Salman Khan and Rao Muhammad Anwer},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.105},
  url = {https://doi.org/10.18653/v1/2025.findings-naacl.105}
}

References

59 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabicaqa: A comprehensive dataset for arabic question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2049–2059.
  2. Google AI. 2023. Gemini: A family of highly capable multimodal models.
  3. Husni A Al-Muhtaseb. 2010. Arabic text recognition of printed manuscripts. Efficient recognition of offline printed Arabic text using Hidden Markov Models, Bigram Statistical Language Model, and post-processing. Ph.D. thesis, University of Bradford. find
  4. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031. ↗
  5. Anthropic. 2024. Claude. AI assistant. find
  6. Arar Tawil. 2023. Arabic food 101. https://www.kaggle.com/datasets/araraltawil/ arabic-food-101. ↗
  7. Muhammad Awais, Ali Husain Salem Abdulla Alharthi, Amandeep Kumar, Hisham Cholakkal, and Rao M. Anwer. 2024. Agrogpt: Efficient agricultural vision-language model with expert tuning. arXiv. ↗
  8. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗
  9. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. ↗
  10. Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342. find
  11. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238. ↗
  12. Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In ICML. ↗
  13. Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. 2024. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. arXiv preprint arXiv:2403.10378. ↗
  14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee. find
  15. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394. ↗
  16. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390. ↗
  17. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904–6913. ↗
  18. Heba Hassan, Ahmed El-Mahdy, and Mohamed E Hussein. 2021. Arabic scene text recognition in the deep learning era: Analysis on a novel dataset. IEEE Access. ↗
  19. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic. ↗
  20. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700– 6709.
  21. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840. ↗
  22. Kartik Kuckreja, Muhammad S. Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad S. Khan. 2024. Geochat: Grounded large vision-language model for remote sensing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  23. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  24. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), pages 19730–19742. PMLR. ↗
  25. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), pages 12888–12900. PMLR. ↗
  26. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. ↗
  27. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740–755. Springer. ↗
  28. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. find
  29. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS. ↗
  30. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer.
  31. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. ↗
  32. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). find
  33. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks. ↗
  34. Sabri A Mahmoud, Irfan Ahmad, Wasfi G Al-Khatib, Mohammad Alshayeb, Mohammad Tanvir Parvez, Volker Märgner, and Gernot A Fink. 2014. Khatt: An open arabic offline handwritten text database. Pattern Recognition, 47(3):1096–1112. ↗
  35. Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263– 2279, Dublin, Ireland. Association for Computational Linguistics.
  36. Mohammad-Alfaifi. n.d. Github - mohammad-alfaifi/arab-celeb-dataset. https://github.com/ mohammad-alfaifi/arab-celeb-dataset. Accessed: 2024-10-15. ↗
  37. Rayyan Najam and Safiullah Faizullah. 2024. Historical arabic handwritten text recognition dataset. ↗
  38. OpenAI. 2024. Gpt-4o model. https://openai.com. Accessed: 2024-10-14. ↗
  39. Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. 2023. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3170–3180. find
  40. PATD. 2024. Printed arabic text database for recognition systems. http://www.inf.u-szeged.hu/ patd/. ↗
  41. Pexel. 2024. Pexel: The best free stock photos, royalty-free images and videos shared by creators. https://www.pexels.com/. ↗
  42. Pinterest. 2024. Pinterest platform. https://www.pinterest.com/. xAI. 2024. xai. grok-1.5 vision preview. https://x. ai/blog/grok-1.5v. ↗
  43. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748–8763. PMLR. ↗
  44. Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706. ↗
  45. Hanoona Rasheed, Muhammad Maaz, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. 2025. Palo: A large multilingual multimodal language model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). find
  46. Eric Sven Ristad and Peter N Yianilos. 1998. Learning string-edit distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(5):522–532. ↗
  47. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967. ↗
  48. Rana SM Saad, Randa I Elanwar, NS Abdel Kader, Samia Mashali, and Margrit Betke. 2016. Bce-arabic-v1 dataset: Towards interpreting arabic document images for people with visual impairments. In Proceedings of the 9th ACM International Conference on Pervasive Technologies Related to Assistive Environments, pages 1–8. find
  49. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  50. Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, and Can Huang. 2024. Mtvqa: Benchmarking multilingual text-centric visual question answering. ↗
  51. Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. 2024. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411. ↗
  52. Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2023. End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2733–2743. find
  53. Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, jiayi lei, Quanfeng Lu, Peng Gao, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2024. MMT-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. In Proceedings of the International Conference on Machine Learning (ICML). find
  54. YouTube. 2024. https://www.youtube.com/. Accessed: 2024-10-01. ↗
  55. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  56. Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2024b. Pangea: A fully open multilingual multimodal llm for 39 languages. arXiv preprint arXiv:2410.16153. ↗
  57. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024c. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813. ↗
  58. Wikipedia. 2024. Wikipedia the free encyclopedia. https://www.wikipedia.org/. ↗
  59. Yue Wu and Prem Natarajan. 2017. Self-organized text detection with minimal post-processing via border learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghaboura-2025-camel-bench-comprehensive-arabic-lmm/