Elamur Research⌕
☰
home / papers / 2026 / How Foundation Models Behave for Arabic Image Captioning?

How Foundation Models Behave for Arabic Image Captioning?

Paper · 2026 · Khaoula Dahimi, Amel Belabbaci, Hadda Cherroun et al.

Summary

The paper asks how well current multimodal foundation models write image captions in Arabic, a morphologically rich language with few resources. The authors also ask whether standard metrics can judge such captions. Their motivation is educational: machine captions could serve as reference answers when grading learners' descriptions of pictures. They tested four model families (Gemini, Gemma, LLaMA and the Arabic-centric Fanar) on two datasets. Flickr8k-Arabic is translated from English, while JEEM was collected natively and covers Arabic dialects. All outputs were normalized and then scored with a broad battery of metrics. These included n-gram overlap, embedding similarity, image–text alignment, LLM judges and hallucination detectors. Model quality varied considerably, and the metric families often disagreed about which captions were good. The authors take this as evidence that multimodal architectures need Arabic-specific tuning, and that a single metric can mislead in low-resource evaluation. The work matters because it maps how reliable off-the-shelf vision-language models are as baselines outside English.

Key points

  • Models compared: Gemini, Gemma, LLaMA and Fanar. This pits general multilingual systems against an Arabic-focused one.
  • Datasets: Flickr8k-Arabic (translated, MSA captions) and JEEM (native, four Arabic dialects).
  • Metrics follow the Sarto et al. (2025) taxonomy:
    • rule-based: BLEU, METEOR
    • learnable: BERTScore
    • visually grounded: TIGEr, CLIPScore (with weight w = 2.5)
    • LLM-based: CLAIR, FLEUR
    • hallucination-oriented: CHAIR, ALOHa
  • Arabic captioning resources fall into two groups:
    • translation-based: Arabic Flickr8k, Multilingual COCO
    • natively collected: JEEM, FutureBeeAI, ImageEval 2025
  • The study has two goals:
    • test whether model captions are consistent enough to act as grading references in education;
    • test whether existing metrics capture caption quality in that setting.

Interesting ideas

  • I1. Score every output with several metric families built on different foundations (surface overlap, embeddings, grounding, LLM judgment), so their disagreements become visible rather than trusting one number.
  • I2. In ALOHa-style hallucination checks, an LLM pulls object mentions out of the output and matches them semantically against what is actually grounded, using an open vocabulary instead of a fixed list.
  • I3. FLEUR smooths an LLM judge's score by weighting the probabilities of each digit token, instead of taking the single sampled number. This yields steadier, more interpretable ratings.
  • I4. Placing a language-specific model (Fanar) next to general multilingual ones tests directly whether specialized training pays off.
  • I5. Separating translated datasets from natively collected ones exposes how provenance shapes benchmark results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation reports could pair exact structural match with looser semantic-similarity scores, so we can see where a change helps one view but hurts another.
  • I2 (back): Every entity mentioned in generated text could be extracted and checked against the world graph it came from, flagging invented entities as compression or generation errors.
  • I3 (learn): LLM teacher tiers that rate parses or frames could report probability-weighted scores, giving calibrated confidence for filtering silver data.
  • I4 (learn): Witness-parser ensembles could deliberately mix specialized and general parsers, then measure which kind of witness actually improves agreement-based silver labels.
  • I5 (tree): Treebanks and silver corpora could carry a provenance tag (native vs converted or translated), so parser scores are reported per origin and conversion artifacts do not distort accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Amar Telidji University of LaghouatLaghouat · Algeria8 works · 121 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksVision & language · 811 works

Cite

Cite as Khaoula Dahimi, Amel Belabbaci, Hadda Cherroun and Abdelhamid Haouhat. 2026. How Foundation Models Behave for Arabic Image Captioning?. The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3bhwcpon3fv5

@inproceedings{dahimi2026how,
  title = {{How Foundation Models Behave for Arabic Image Captioning?}},
  author = {Khaoula Dahimi, Amel Belabbaci, Hadda Cherroun and Abdelhamid Haouhat},
  booktitle = {The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks},
  year = {2026},
  doi = {10.63317/3bhwcpon3fv5},
  url = {https://doi.org/10.63317/3bhwcpon3fv5}
}

References

36 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Imad Afyouni, Imtinan Azhar, and Ashraf Elnagar. 2021. Aracap: A hybrid deep learning architecture for arabic image captioning. Procedia Computer Science, 189:382–389. find
  2. Rasha Saleh Al-Malki and Arwa Yousuf Al-Aama. 2023. Arabic captioning for images of clothing using deep learning. Sensors, 23(8):3783. ↗
  3. Ashwaq Alsayed, Thamir M. Qadah, and Muhammad Arif. 2023. A performance analysis of transformer-based deep learning models for arabic image captioning. Journal of King Saud University - Computer and Information Sciences, 35(9):101750. ↗
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, page 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. Ahlam Bashiti, Alaa Aljabari, Hadi Khaled Hamoud, Md.
  5. Rafiul Biswas, Bilal Mohammed Shalash, Mustafa Jarrar, Fadi Zaraket, George Mikros, Ehsaneddin Asgari, and Wajdi Zaghouani. 2025. Imageeval 2025: The first arabic image captioning shared task. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, page 376–389, Suzhou, China. Association for Computational Linguistics.
  6. David M. Chan, Suzanne Petryk, Joseph E. Gonzalez, Trevor Darrell, and John Canny. 2023. Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 13638–13646, Singapore. Association for Computational Linguistics. find
  7. Xiaofu Chen, Israfel Salazar, and Yova Kementchedjhieva. 2025. Specs: Specificity-enhanced clip-score for long image caption evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9406–9418. ↗
  8. Khaled Bachir Delassi, Lakhdar Zeggane, Hadda Cherroun, Abdelhamid Haouhat, and Kaoutar Bouzouad. 2025. Vqa support to arabic language learning educational tool. ↗
  9. Yashwant Dongare, Bhalchandra M. Hardas, Rashmita Srinivasan, Vidula Meshram, Mithun G. Aush, and Atul Kulkarni. 2024. Deep neural networks for automated image captioning to improve accessibility for visually impaired users. International Journal of Intelligent Systems and Applications in Engineering, 12(2s):267–281. find
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of models.
  11. Passant Elchafei and Amany Fashwan. 2025. Multimodal arabic captioning with interpretable visual concept integration. (arXiv:2510.03295). ArXiv:2510.03295 [cs]. ↗
  12. Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, and Daniel Asmar. 2020. Resources and end-to-end neural network models for arabic image captioning:. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, page 233–241, Valletta, Malta. SCITEPRESS - Science and Technology Publications. find
  13. Jonathan Emami, Pierre Nugues, Ashraf Elnagar, and Imad Afyouni. 2022. Arabic image captioning using pre-training of deep bidirectional transformers. In Proceedings of the 15th International Conference on Natural Language Generation, page 40–51, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
  14. FutureBee AI. 2025. Arabic image caption dataset. https://www.futurebeeai.com/dataset/multi-modal-dataset/ arabic-image-caption-dataset. Accessed: February 7, 2026. ↗
  15. Michael Hanna and Ondřej Bojar. 2021. A fine-grained analysis of bertscore. In Proceedings of the Sixth Conference on Machine Translation, pages 507–517. ↗
  16. Abdelhamid Haouhat, Slimane Bellaouar, Attia Nehar, Hadda Cherroun, and Ahmed Abdelali. 2025. Arabic multimodal machine learning: Datasets, applications, approaches, and challenges. arXiv e-prints, pages arXiv–2508.
  17. Mohammad Nehal Hasnine, Brendan Flanagan, Gokhan Akcapinar, Hiroaki Ogata, Kousuke Mouri, and Noriko Uosaki. 2019. Vocabulary Learning Support System Based on Automatic Image Captioning Technology, volume 11587 of Lecture Notes in Computer Science, page 346– 358. Springer International Publishing, Cham. find
  18. Hani Hejazi and Khaled Shaalan. 2021. Deep learning for arabic image captioning: A comparative study of main factors and preprocessing recommendations. International Journal of Advanced Computer Science and Applications, 12(11). ↗
  19. Khalil Hennara, Muhammad Hreden, Mohamed Motaism Hamed, Zeina Aldallal, Sara Chrouf, and Safwan AlModhayan. 2025. Mutarjim: Advancing bidirectional arabic-english translation with a small language model. ↗
  20. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2022. Clipscore: A reference-free evaluation metric for image captioning. (arXiv:2104.08718). ArXiv:2104.08718 [cs]. ↗
  21. Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. 2019. Tiger: Text-to-image grounding for image caption evaluation. (arXiv:1909.02050). ArXiv:1909.02050 [cs]. ↗
  22. Romrawin JP. 2025. Multilingual coco dataset. https://huggingface.co/datasets/romrawinjp/multilingual-coco. Accessed: February 7, 2026. ↗
  23. Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Kementchedjhieva. 2025. Jeem: Vision-language understanding (arXiv:2503.21910). in four arabic dialects. ArXiv:2503.21910 [cs]. ↗
  24. Yebin Lee, Imseong Park, and Myungjoo Kang. 2024. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. (arXiv:2406.06004). ArXiv:2406.06004 [cs]. ↗
  25. Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. 2024. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478. ↗
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, page 311, Philadelphia, Pennsylvania. Association for Computational Linguistics.
  27. Suzanne Petryk, David M. Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E. Gonzalez, and Trevor Darrell. 2024. Aloha: A new measure for hallucination in captioning models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), page 342–357, Mexico City, Mexico. Association for Computational Linguistics. ↗
  28. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819. ↗
  29. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2019. Object hallucination in image captioning. (arXiv:1809.02156). ArXiv:1809.02156 [cs]. ↗
  30. Sara Sarto, Marcella Cornia, Rita Cucchiara, et al. 2025. Image captioning evaluation in the age of multimodal llms: Challenges and future perspectives. In Proceedings of the 34th International Joint Conference on Artificial Intelligence. ↗
  31. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025a. Fanar: An arabic-centric multimodal generative ai platform. ↗
  32. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  33. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025b. Gemma 3 Technical Report. ↗
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. (arXiv:1904.09675). ArXiv:1904.09675 [cs]. ↗
  35. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592. ↗
  36. Amir Zur, Elisa Kreiss, Karel D’Oosterlinck, Christopher Potts, and Atticus Geiger. 2024. Updating clip to prefer descriptions over captions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20178–20187. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dahimi-2026-how-foundation-models-behave-arabic/