📄 Codezone Research Group at ImageEval Shared-Task 2: Arabic Image Captioning Using BLIP and M2M100: A Two-Stage Translation Approach for ImageEval 2025

Summary

The paper presents a system built for ImageEval 2025's shared task, which focuses on generating Arabic captions for images. Arabic has few image–text resources, so the authors do not train an Arabic captioner. They chain two existing models without any fine-tuning. BLIP-base first writes an English caption using beam search. The best-scoring caption is then translated into Arabic with M2M100-418M. Before scoring, the Arabic output is normalized by stripping diacritics, Tatweel and punctuation and by standardizing whitespace.

On the 752-image test set, the pipeline beats both zero-shot and fine-tuned Qwen 2.5-VL 7B on BLEU. It falls behind them on cosine similarity and on the LLM Judge score. Manual rating found weak cultural relevance and weak completeness. The authors trace the errors to idioms, culture-specific objects and Arabic morphology.

The work shows that a translation-based pipeline is a cheap, workable baseline for a low-resource language. It also shows that such a pipeline tends to match the references word by word without capturing deeper meaning.

Key points

• The ImageEval 2025 dataset is reported as 3,471 images with Arabic captions. The stated splits are 2,718 train, 75 validation and 752 test, which add up to more than 3,471.

• The models are Salesforce/blip-image-captioning-base and facebook/m2m100-418M, run with PyTorch and Transformers. Inference is zero-shot.

• Test results for BLIP+M2M100 versus zero-shot and fine-tuned Qwen 2.5-VL 7B:

  Metric	BLIP+M2M100	Qwen zero-shot	Qwen fine-tuned
  BLEU-1	0.2847	0.0992	0.1698
  Cosine similarity	0.383	0.5577	0.5846
  LLM Judge	15.14	27.11	30.82

• ROUGE is reported as 0.0000 for every system, which suggests a scoring problem in the evaluation.

• Qualitative ratings were cultural relevance 1.10, conciseness 2.03, completeness 1.47 and accuracy 2.03. Captioning plus translation took about 12.27 s per image.

Interesting ideas

• I1. Generate in a high-resource language and then translate, so that each module can be replaced or debugged on its own. The weak point is that the system never learns how native speakers actually describe things.
• I2. Surface-overlap metrics and semantic metrics ranked the systems in opposite orders. Reporting only one family of metrics would therefore have been misleading.
• I3. The output is normalized before scoring (diacritics, elongation characters, punctuation, whitespace). This keeps purely orthographic noise from distorting the comparison with references.
• I4. Captions are rated on separate axes (cultural relevance, conciseness, completeness, accuracy) instead of one overall score. This shows exactly which quality is weak.
• I5. The error taxonomy names idioms with no target equivalent and culture-specific objects as recurring failures.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): Mova could produce English from the meaning graph first and treat translation as a separate, inspectable step, logging where target-language idioms are lost.
• I2 (learn): Mova's evaluation should pair exact-match scores (LAS, lemma accuracy) with meaning-level checks, so that a gain on one is not mistaken for overall progress.
• I3 (tokens): a shared normalizer for apostrophes, quotes, hyphens and case could be applied before comparing tokenizer and lemma output with treebank gold.
• I4 (answer): QA and school-test answers could be graded separately for correctness, completeness and brevity, to tell whether the reader missed a fact or only phrased the answer badly.
• I5 (frames): idioms could be detected before frame building and mapped to their figurative frame, so they are not read literally.