Elamur Research⌕
☰
home / papers / 2025 / ImageEval 2025: The First Arabic Image Captioning Shared Task

ImageEval 2025: The First Arabic Image Captioning Shared Task

Paper · 2025 · Ahlam Bashiti, Alaa Aljabari, Hadi Hamoud et al.

Summary

ImageEval 2025 is the first shared task built around generating Arabic captions for images. Arabic multimodal work has been held back by a lack of native caption data, and most existing sets are translated from English, which loses cultural detail. The organizers addressed this with two linked subtasks. The first was a collaborative datathon in which participants wrote Arabic captions by hand, without generative AI and with only minimal context about each image. The second was a competition in which teams built and evaluated captioning models. Submissions were scored on Codabench with three kinds of evaluation: automatic overlap and similarity metrics, judgments from an LLM, and a human rubric applied to a small sample. The best scores were 65.5 cosine similarity in Subtask 1 and 60.0 in Subtask 2. These results show progress, but the task is clearly not solved: cultural grounding, rich morphology and dialect variation remain hard. All data, baselines and scoring tools are released openly, giving Arabic vision-language research a common benchmark.

Key points

  • 44 teams registered, 8 submitted in the test phase, and there were 111 valid submissions; all 8 system description papers were accepted.
  • The top results were a cosine similarity of 65.5 in Subtask 1 and 60.0 in Subtask 2.
  • Automatic metrics included BLEU, cosine similarity, TF-IDF over 1–4-grams, Jaccard and Lin similarity. Humans rated about 5% of the test data on four criteria, such as Cultural Relevance, using a 1–4 scale.
  • Participant systems mostly fine-tuned pretrained multilingual vision-language models, often with LoRA adapters. Other approaches included:
    • captioning in English and then translating into Arabic;
    • a region-aware transformer;
    • augmentation with caption validation;
    • multi-stage pipelines that describe the image first and then refine the description into a caption.
  • A stated limitation is that the images focus on Palestinian cultural contexts. A GPT-5 mini example produced a generic coastal-city caption, whereas the human caption named the specific mosque in Acre.

Interesting ideas

  • I1. Gold captions were written by people who were explicitly barred from using generative tools, so the reference set stays independent of the model outputs it will judge.
  • I2. Scoring was triangulated across cheap overlap metrics, an LLM judge and a small human-rated sample with a fixed rubric, so each layer catches failures the others miss.
  • I3. The paper contrasts an accurate but generic caption with a culturally specific one, which separates "correct" from "informative" as distinct failure modes.
  • I4. Some teams wrote a detailed description first and only then condensed it into a short caption, splitting content selection from surface brevity.
  • I5. Lin similarity scores two texts through the information content of their least common subsumer in a taxonomy, which rewards near-synonyms that plain overlap metrics ignore.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Keep a hand-built gold slice that no LLM teacher tier has touched, and use it as the uncontaminated control when scoring silver data.
  • I2 (learn): Pair automatic tree and answer metrics with an LLM-judge pass and a periodic human audit of about 5% of items using a short rubric.
  • I3 (back): Generation from the meaning graph could be checked for whether it names the specific entities held in the world, rather than defaulting to category words.
  • I4 (back): The graph-to-text step could first verbalize the full graph and then compress it, making each dropped fact an explicit, explainable decision.
  • I5 (world): Use WordNet-based Lin similarity over the ontology to compare entities or answers, and award partial credit when a reader names a hypernym or sibling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations🎓Northwestern University in QatarDoha · Qatar24 works · 148 citations🔬Arab Center for Research and Policy StudiesDoha · Qatar8 works · 69 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Ahlam Bashiti, Alaa Aljabari, Hadi Hamoud, Md. Rafiul Biswas, Bilal Shalash, Mustafa Jarrar, Fadi Zaraket, George Mikros, Ehsaneddin Asgari and Wajdi Zaghouani. 2025. ImageEval 2025: The First Arabic Image Captioning Shared Task. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.52

@inproceedings{bashiti2025imageeval,
  title = {{ImageEval 2025: The First Arabic Image Captioning Shared Task}},
  author = {Ahlam Bashiti, Alaa Aljabari, Hadi Hamoud, Md. Rafiul Biswas, Bilal Shalash, Mustafa Jarrar, Fadi Zaraket, George Mikros, Ehsaneddin Asgari and Wajdi Zaghouani},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.52},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.52}
}

References

62 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Limitation Imad Afyouni, Imtinan Azhar, and Ashraf Elnagar. 2021. Aracap: A hybrid deep learning architecture for arabic image captioning. In Procedia Computer Science, pages 382–389. This study is limited to Palestinian cultural representation and does not cover other Arabic-speaking regions. The dataset captions are exclusively in MSA and do not include regional dialects. Therefore, while suitable for training models on Pales-Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. Nadi 2023: The fourth nuanced arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics.
  2. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. Nadi 2024: The fifth nuanced arabic dialect identification shared task. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 709– 728, Bangkok, Thailand. Association for Computational Linguistics. Muhammad Abu Horaira, Farhan Amin, Sakibul Hasan, Md.
  3. Tanvir Ahammed Shawon, and Muhammad Ibrahim Khan. 2025. Phantomtroupe at imageeval shared task: Multimodal arabic image captioning through translation-based fine-tuning of llm models. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  4. Diyam Akra, Tymaa Hammouda, and Mustafa Jarrar. 2025. Quranmorph: Morphologically annotated quranic corpus. Technical report, Birzeit University. ↗
  5. Emran Al-Buraihy and Dan Wang. 2024. Enhancing cross-lingual image description: A multimodal approach for semantic relevance and stylistic alignment. Computers, Materials & Continua. ↗
  6. Emran Al-Buraihy, Dan Wang, Tariq Hussain, R. Attar, A. Alzubi, Khalid Zaman, and Zengkang Gan. 2025. Aratraditions10k bridging cultures with a comprehensive dataset for enhanced cross lingual image annotation retrieval and tagging. Scientific Reports. ↗
  7. Rabee Al-Qasem and Mohannad Hendi. 2025. Impactai at imageeval 2025 shared task: Region-aware transformers for arabic image captioning—a case study on the palestinian narrative. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  8. Mohammed Alkhanafseh, Ola Surakhi, and Abdallah Abedaljalill. 2025. Bzu-aum at imageeval 2025: An arabic image captioning dataset for conflict narratives with human annotation. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  9. Ashwaq Alsayed, Thamir M. Qadah, and Muhammad Arif. 2023. A performance analysis of transformer-based deep learning models for arabic image captioning. Journal of King Saud University – Computer and Information Sciences, 35(8):101684. ↗
  10. Anfal Attai and Ashraf Elnagar. 2020. A survey on arabic image captioning systems using deep learning models. International Conference on Innovations in Information Technology. find
  11. Israa Al Badarneh, Rana Husni Al Mahmoud, Bassam H. Hammo, and Omar S. Al-Kadi. 2025. Attention-based transformer model for arabic image captioning. Neural Computing and Applications. find
  12. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. ↗
  13. Siddharth Betala and Ishan Chokshi. 2024. Brotherhood at WMT 2024: Leveraging LLM-generated contextual conversations for cross-lingual image captioning. arXiv preprint arXiv:2409.15052. ↗
  14. Abdulkadir Shehu Bichi, Ismail Dauda Abubakar, Fatima Muhammad Adam, Aminu Musa, Auwal Umar Ahmed, Abubakar Ibrahim, Khadija Salihu Aua, Aisha Mustapha Ahmed, and Mahmud Said Ahmed. 2025. Codezone research group at imageeval 2025 shared task: Arabic image captioning using BLIP and M2M100—a two-stage translation approach. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  15. Suhyun Cho and Hayoung Oh. 2023. Generalized image captioning for multilingual support. Applied Sciences. ↗
  16. Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C. Lawrence Zitnick. 2015. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467. ↗
  17. Samar Elbedwehy and Tamer Mohammed Ibrahim Medhat. 2023. Improved arabic image captioning model using feature concatenation with pre-trained word embedding. Neural Computing and Applications, 35:19051–19067. ↗
  18. Passant Elchafei and Amany Fashwan. 2025. Vlcap at imageeval 2025 shared task: Multimodal arabic captioning with interpretable visual concept integration. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  19. Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, and Daniel Asmar. 2020. Resources and end-to-end neural network models for arabic image captioning. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VIS-APP), pages 233–241. find
  20. Jonathan Emami, P. Nugues, A. Elnagar, and Imad Afyouni. 2022a. Arabic image captioning using pre-training of deep bidirectional transformers. International Conference on Natural Language Generation.
  21. Jonathan Emami, Pierre Nugues, Ashraf Elnagar, and Imad Afyouni. 2022b. Arabic image captioning using pre-training of deep bidirectional transformers. In Proceedings of the 15th International Conference on Natural Language Generation.
  22. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In European Conference on Computer Vision, pages 15– 29. Springer. find
  23. Rana Gaber, Seif Eldin Amgad, Ahmed Sherif Nasri, Mohamed Ibrahim Ragab, and Ensaf Hussein Mohamed. 2025. NU_Internship team at imageeval 2025: From zero-shot to ensembles—enhancing grounded arabic image captioning. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  24. Lianli Gao, Xiangpeng Li, Jingkuan Song, and Heng Tao Shen. 2020. Hierarchical LSTMs with adaptive attention for visual captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:1112–1131. ↗
  25. Hani Hejazi and Khaled Shaalan. 2021. Deep learning for arabic image captioning: A comparative study of main factors and preprocessing recommendations. International Journal of Advanced Computer Science and Applications. ↗
  26. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal pivots for image caption translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Md. Zakir Hossain, Ferdous Sohel, Mohd. find
  27. Fairuz Shiratuddin, and Hamid Laga. 2018. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys, 51:1–36. Md. Zakir Hossain, Ferdous Sohel, Mohd. find
  28. Fairuz Shiratuddin, and Hamid Laga. 2019. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CSUR), 51(6):1–36. find
  29. George Ibrahim, Rita Ramos, and Yova Kementchedjhieva. 2025. Concap: Seeing beyond english with concepts retrieval-augmented captioning. arXiv.org. ↗
  30. Mustafa Jarrar. 2021. The arabic ontology: An arabic wordnet with ontologically clean content. Applied Ontology, 16(1):1–26. ↗
  31. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa’ Omar. 2023a. Wojoodner 2023: The first arabic named entity recognition shared task. In Proceedings of ArabicNLP 2023, pages 748–758, Singapore (Hybrid). Association for Computational Linguistics.
  32. Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2024. Wojoodner 2024: The second arabic named entity recognition shared task. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 847–857, Bangkok, Thailand. Association for Computational Linguistics.
  33. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023b. Lisan: Yemeni, iraqi, libyan, and sudanese arabic dialect corpora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA), pages 1–7, Egypt. IEEE. find
  34. Karima Kadaoui, Hanin Atwany, Hamdan Hamid Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Kementchedjhieva. 2025. Jeem: Vision-language understanding in four arabic dialects. arXiv.org. ↗
  35. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. Arabicnlu 2024: The first arabic natural language understanding shared task. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 361–371, Bangkok, Thailand. Association for Computational Linguistics.
  36. Nikhil Gopal Khodave and Prathamesh S. Powar. 2025. Survey on multimodal image captioning approaches: Addressing contextual understanding, cross-dataset generalization, and multilingual captioning. International Journal of Advanced Research in Science, Communication and Technology. ↗
  37. Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2891–2903. ↗
  38. Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. 2019. COCO-CN for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia, 21:2347–2360. ↗
  39. Maofu Liu, Lingjun Li, Huijun Hu, Weili Guan, and Jing Tian. 2020. Image caption generation with dual attention mechanism. Information Processing & Management, 57:102178. ↗
  40. Yue Ming, Nannan Hu, Chunxiao Fan, Fan Feng, Jiang-wan Zhou, and Hui Yu. 2022. Visuals to text: A comprehensive review on automatic image captioning. IEEE/CAA Journal of Automatica Sinica, 9:1339– 1365. ↗
  41. Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Alcoba Inciarte, and M. Abdul-Mageed. 2023a. Violet: A vision-language model for arabic image captioning with gemini decoder. ARABICNLP.
  42. Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Alcoba Inciarte, and Muhammad Abdul-Mageed. 2023b. Violet: A vision-language model for arabic image captioning with gemini decoder. arXiv preprint arXiv:2311.08844. ↗
  43. Amal Nayouf, Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian arabic dialects with morphological annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of EMNLP 2023, pages 12–23, Singapore. Association for Computational Linguistics. ↗
  44. Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. 2023. Improving multimodal datasets with image captioning. arXiv preprint arXiv:2307.10350. ↗
  45. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in Neural Information Processing Systems, 24. find
  46. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. Association for Computational Linguistics.
  47. Hashem Parvin, Ahmad Reza Naghsh-Nilchi, and Hossein Mahvash Mohammadi. 2023. Transformer-based local-global guidance for image captioning. Expert Systems with Applications, 223:119774. ↗
  48. Sabri Monaf Sabri. 2021. Arabic image captioning using deep learning with attention. Master’s thesis, University of Georgia. find
  49. Mariam Saeed, Sarah Elshabrawy, Abdelrahman Ha-grass, Mazen Yasser, and Ayman Khalafallah. 2025. Averroes at imageeval 2025 shared task: Advancing arabic image captioning with augmentation and two-stage generation. In Proceedings of the Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
  50. Mala Saraswat, Challa Vivekananda Reddy, and Garan-dal Yashwanth Singh. 2024. Image captioning using NLP. In 2024 First International Conference on Technological Innovations and Advance Computing (TIACOMP), pages 549–553. IEEE. find
  51. Naeha Sharif, Lyndon White, Mohammed Bennamoun, Wei Liu, and Syed Afaq Ali Shah. 2020. Wembsim: A simple yet effective metric for image captioning. arXiv. find
  52. Zijie Song, Zhenzhen Hu, Yuanen Zhou, Ye Zhao, Richang Hong, and Meng Wang. 2023. Embedded heterogeneous attention transformer for cross-lingual image captioning. IEEE Transactions on Multimedia, 26:9008–9020. ↗
  53. Chenggang Yan, Yiming Hao, Liang Li, Jian Yin, An-An Liu, Zhendong Mao, Zhenyu Chen, and Xingyu Gao. 2021. Task-adaptive attention for image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 32:43–51. ↗
  54. Sarah Yassine, Sara Mahrous, and Rawan Sous. 2025. Azlu at imageeval 2025: Bridging linguistic and cultural gaps in arabic image captioning. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  55. Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. 2019. Multimodal transformer with multi-view visual representation for image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 30:4467–4480. ↗
  56. Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, and Feng Wu. 2022. Context-aware visual policy network for fine-grained image captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:710–722. find
  57. Jing Zhang, Dan Guo, Xun Yang, Peipei Song, and Meng Wang. 2023. Visual-linguistic-stylistic triple reward for cross-lingual image captioning. ACM Transactions on Multimedia Computing, Communications and Applications, 20:1–23. ↗
  58. Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, and Chen Ma. 2025. Crowd comparative reasoning: Unlocking comprehensive evaluations for LLM-as-a-judge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5059–5074, Vienna, Austria. Association for Computational Linguistics. ↗
  59. Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. 2023. From show to tell: A survey on deep learning-based image captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:539–559. ↗
  60. Akash Verma, Arun Kumar Yadav, Mohit Kumar, and Divakar Yadav. 2023. Automatic image caption generation using deep learning. Multimedia Tools and Applications, 83:5309–5325. ↗
  61. Junbo Wang, Wei Wang, Liang Wang, Zhiyong Wang, Dagan Feng, and Tieniu Tan. 2020. Learning visual relationship and context-aware attention for image captioning. Pattern Recognition, 98. ↗
  62. Tiantao Xian, Zhixin Li, Canlong Zhang, and Huifang Ma. 2022. Dual global enhanced transformer for image captioning. Neural Networks, 148:129–141. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bashiti-2025-imageeval-2025-first-arabic-image/