Elamur Research⌕
☰
home / papers / 2026 / JEEM: Vision-Language Understanding in Four Arabic Dialects

JEEM: Vision-Language Understanding in Four Arabic Dialects

Paper · 2026 · Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali et al.

Summary

JEEM addresses a gap in evaluating vision-language models (VLMs) for Arabic. Existing Arabic multimodal data mostly comes from translated or relabeled Western images, and it is written in MSA or covers only one dialect. The authors built a benchmark with images from Jordan, the Emirates, Egypt and Morocco, one representative dialect per major dialect region. Native-speaker annotators wrote detailed captions in both dialect and MSA, plus question–answer pairs in dialect. The annotation protocol was modeled on a smart assistant describing a photo to a user who cannot see it. Six models were tested on captioning and VQA: five open-source Arabic VLMs (Peacock, Maya, AIN, Palo, AyaV) and GPT-4o. Evaluation combined n-gram metrics, GPT-4 as a judge, DCScore, human ratings and a fine-grained error taxonomy. The Arabic models lag well behind on visual grounding and on producing dialect. GPT-4o is strongest, but its dialect quality is uneven and its visual understanding is still limited. Emirati and Moroccan are the hardest varieties. The work shows that culturally and dialectally diverse evaluation exposes weaknesses that standard benchmarks hide.

Key points

  • The dataset has 2,196 images: Egypt 863, Jordan 606, Morocco 577 and Emirates 150. It is released on Hugging Face (toloka/JEEM), with annotations under the MIT license.
  • Images come from Wikimedia Commons, Creative Commons Flickr and contributors' personal photos. After annotation they were grouped into 13 themes (food, transport, places, etc.) with the help of GPT-4o mini.
  • Annotators were selected through a qualification task, giving 10, 8, 10 and 9 annotators per country. In addition, 25 culturally distinctive images per country were annotated by all four teams.
  • The error analysis is striking. Maya hallucinates in 73% of the analysed outputs. AyaV invents words (14%) and mixes dialects (16%). GPT-4o makes the fewest errors of leaving cultural details underspecified (11%) and of misinterpretation (10%).
  • On the Emirati set, GPT-4o and Maya fall back to fully MSA output 81% of the time. The figure is 18% for Jordan and 0% for Egypt and Morocco.

Interesting ideas

  • I1. Question writers saw only the caption, never the image, so their questions reflect a real gap in information rather than what is visually obvious; unanswerable questions are flagged explicitly.
  • I2. Annotators wrote in dialect first and only then in MSA, so the spontaneous variety was not shaped by the standard one.
  • I3. The same small pool of images was annotated by every regional team, which allows a direct comparison of how different communities describe identical content.
  • I4. Model failures were sorted into separate families: linguistic (made-up words, dialect mixing), visual grounding (hallucination, wrong counts), cultural (underspecified or misread items) and generation (degeneration, irrelevant content).
  • I5. Automatic metrics were checked against human judgments before being used to make recommendations, and the authors note the risk of a GPT-4 judge favoring GPT-4o.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Questions for fable and tale QA could be generated from a summary alone, which yields natural "what was left unsaid" probes, and the reader must learn to say when the text does not support an answer.
  • I2 (tree): When building silver or converted treebanks for nonstandard English varieties, collecting the informal text before its normalized counterpart would keep dialect syntax authentic for UD conversion testing.
  • I3 (world): Having several witnesses describe one fixed scene could show which entities and states are culturally salient, and help decide what the world model should treat as default versus noteworthy.
  • I4 (learn): Evaluation reports could break errors down by pipeline stage and error type in the same way, instead of a single score, making regressions traceable to tokens, tree, frames or world.
  • I5 (learn): LLM teacher tiers used for grading should be calibrated against a human-checked subset, and judges from the same model family as the system being graded should be treated as a possible bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki and Yova Kementchedjhieva. 2026. JEEM: Vision-Language Understanding in Four Arabic Dialects. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.18

@inproceedings{kadaoui2026jeem,
  title = {{JEEM: Vision-Language Understanding in Four Arabic Dialects}},
  author = {Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki and Yova Kementchedjhieva},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.18},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.18}
}

References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Huda A. Al-muzaini, Tasniem N. Al-yahya, and Hafida Benhidour. 2018. Automatic arabic image captioning using rnn-lstm-based language model and cnn. International Journal of Advanced Computer Science and Applications, 9(6). find
  2. Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, SM Uddin, Shayekh Bin Islam, et al. 2024. Maya: An instruction finetuned multilingual multimodal model. arXiv preprint arXiv:2412.07112. ↗
  3. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of Arabic multimodal large language models and benchmarks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12753–12776, Bangkok, Thailand. Association for Computational Linguistics. ↗
  4. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged Al-shaibani. 2024. CIDAR: Culturally relevant instruction dataset for Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12878–12901, https://wikimedia.org/Licensing/ https://flickrhelp.com/creativecommons/ 6 https://opensource.org/license/mit Bangkok, Thailand. Association for Computational Linguistics. ↗
  5. Zaid Alyafeai, Maged S. Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi, and Ali Fadel. 2023. Taqyim: Evaluating arabic nlp tasks using chatgpt models. Preprint, arXiv:2306.16322. ↗
  6. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726. ↗
  7. Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, EunJeong Hwang, and Vered Shwartz. 2024. From local concepts to universals: Evaluating the multicultural understanding of vision-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6763–6782. find
  8. Kyle Buettner and Adriana Kovashka. 2024. Quantifying the gaps between translation and native perception in training for multimodal, multilingual retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5863–5870, Miami, Florida, USA. Association for Computational Linguistics. find
  9. Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023. Maxm: Towards multilingual visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2667–2682. find
  10. Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486. ↗
  11. Cohere. 2025. Aya vision: Expanding the worlds ai can see. Accessed: March 21, 2025. find
  12. Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya. 2020. A unified framework for multilingual and code-mixed visual question answering. In Proceedings of the 1st conference of the Asia-Pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing, pages 900–913. find
  13. Laura Gustafson, Megan Richards, Melissa Hall, Caner Hazirbas, Diane Bouchacourt, and Mark Ibrahim. 2023. Exploring why object recognition performance degrades across income levels and geographies with factor annotations. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  14. Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  15. Ahmed Heakl, Sara Ghaboura, Omkar Thawkar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, and Salman Khan. 2025. Ain: The arabic inclusive large multimodal model. arXiv preprint arXiv:2502.00094. ↗
  16. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  17. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853–899. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  19. Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. 2019. Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 52–59. find
  20. Vasu Jindal. 2017. A deep learning approach for arabic caption generation using roots-words. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1). find
  21. Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem M Hajj, and Daniel C Asmar. 2020. Resources and end-to-end neural network models for arabic image captioning. In VISIGRAPP (5: VIS-APP), pages 233–241. find
  22. Ákos Kádár, Desmond Elliott, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Lessons learned in multilingual grounded language learning. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 402–412, Brussels, Belgium. Association for Computational Linguistics. find
  23. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question. Advances in neural information processing systems, 28. find
  24. Sarah M Kamel, Shimaa I Hassan, and Lamiaa Elrefaei. 2023. Vaqa: Visual arabic question answering. Arabian Journal for Science and engineering, 48(8):10803–10823. ↗
  25. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics.
  26. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗
  27. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  28. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing. ↗
  29. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  30. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
  31. LlamaTeam. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  32. Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Inciarte, and Muhammad Abdul-Mageed. 2023. Violet: A vision-language model for Arabic image captioning with gemini decoder. In Proceedings of ArabicNLP 2023, pages 1–11, Singapore (Hybrid). Association for Computational Linguistics.
  33. Team OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  35. Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022. xgqa: Cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511. find
  36. Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, and Ibrahim Alabdulmohsin. 2024. No filter: Cultural and socioeconomic diversity in contrastive vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  37. Hanoona Rasheed, Muhammad Maaz, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. 2025. Palo: A polyglot large multimodal model for 5b people. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 1745–1754. ↗
  38. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967. ↗
  39. Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. 2024. Mtvqa: Benchmarking multilingual text-centric visual question answering. arXiv preprint arXiv:2405.11985. ↗
  40. Tony Cheng Tong, Sirui He, Zhiwen Shao, and Dit-Yan Yeung. 2024. G-veval: A versatile metric for evaluating image and video captions using gpt-4o. Preprint, arXiv:2412.13647. ↗
  41. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↗
  42. Rasha Mualla and Jafar Alkheir. 2018. Development of an arabic image description system. Int. J. Comput. Sci. Trends Technol, 6:205–213. find
  43. Richard E Nisbett and Takahiko Masuda. 2013. Culture and point of view. In Biological and cultural bases of human inference, pages 49–70. Psychology Press. find
  44. Yuiga Wada, Kanta Kaneda, Daichi Saito, and Komei Sugiura. 2024. Polos: Multimodal metric learning from human feedback for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13559– 13568. find
  45. Yuxuan Wang, Yijun Liu, Fei Yu, Chen Huang, Kexin Li, Zhiguo Wan, and Wanxiang Che. 2024. 4. Use everyday language: Use ordinary informal language, but feel free to incorporate slang where appropriate. Cvlue: A new benchmark dataset for chinese vision-language understanding evaluation. Preprint, arXiv:2407.01081. ↗
  46. Qinghao Ye, Xianhan Zeng, Fu Li, Chunyuan Li, and Haoqi Fan. 2025. Painting with words: Elevating detailed image captioning with benchmark and alignment learning. Preprint, arXiv:2503.07906. ↗
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kadaoui-2026-jeem-vision-language-understanding-four/