📄 ImpactAi at ImageEval 2025 Shared Task: Region-Aware Transformers for Arabic Image Captioning – A Case Study on the Palestinian Narrative

Summary

The authors compete in the ImageEval 2025 Shared Task, entering its Subtask 2, where systems must produce Arabic captions for photos tied to Palestinian history. The shared-task dataset has 3,471 images, each with one human caption. The images mix crowds, artifacts and black-and-white photos, and many show named people.

Their main system, CRAFT, does not encode the whole picture. It takes 50 object regions found by a pre-trained Faster R-CNN (ResNet-50) and passes them, with their box coordinates, through a small transformer encoder. A transformer decoder that uses the ArabGlossBERT vocabulary then writes the caption with beam search. A second system, AraViT-GPT (a ViT encoder paired with a GPT-2 decoder that was given an Arabic tokenizer), serves as the comparison point.

CRAFT scores best on embedding cosine similarity (56.22 on test). AraViT-GPT is slightly better on BLEU and on the GPT-4o judge. A fine-tuned Qwen 2.5-VL 7B baseline beats both on semantic metrics. The paper shows that region-level input helps with crowded historical scenes, but the model still invents the names of people it cannot verify.

Key points

• The data split is 2,718 training and 753 test images, plus 75 validation images released during the task. No external data was used.

• CRAFT feeds k = 50 ROIs per image (a 50 × 1024 tensor) to the encoder. Choosing k dynamically with the elbow method sometimes lost important objects, and k = 100 brought no gain.

• Grid search over short 10-epoch runs selected 2 layers and 2 heads for both encoder and decoder, with batch size 8. The resulting model has about 132M parameters and was trained on a single T4, with early stopping at epoch 30.

• Test scores were:

  System	Cosine	LLM judge	BLEU-1	ROUGE
  CRAFT	56.22	22.34	19.07	0
  AraViT-GPT	55.46	26.55	21.40	0
  Fine-tuned Qwen 2.5-VL 7B baseline	58.46	30.82	—	—

• On the official leaderboard the team placed 4th on cosine similarity and on the LLM evaluation, and 5th in the human evaluation.

Interesting ideas

• I1. Repairing the text before tokenization: the slash in dates such as "09/1970" became an unknown token, so the authors replaced it with a space, and this cheap fix improved predictions.
• I2. Fixing a fixed budget of input units beat an adaptive clustering choice, because the adaptive method sometimes discarded the very objects that mattered.
• I3. Naming one primary metric before comparing systems, while still reporting the rest, made it explicit why metrics that disagree lead to a particular choice.
• I4. ROUGE came out at zero for every system, including the baseline, which signals a broken measurement rather than equally bad systems.
• I5. The model described the scene correctly but added plausible names that were not in the photo, so correctness at the scene level and at the entity level need separate checks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Audit which characters in dates, fractions and measures fall out of the tokenizer's vocabulary, and normalize them with explicit, logged rules rather than letting them silently turn into unknowns.
• I2 (world): When pruning candidate entities or states from a text's world, prefer a generous fixed cap over aggressive automatic pruning, and test whether pruning drops entities that questions later ask about.
• I3 (learn): For each evaluation, state up front which metric decides, for example correct answers versus tree overlap, so that BLEU-like proxies cannot quietly reverse a decision.
• I4 (learn): Treat a metric that is flat across all systems, including negative controls, as an alarm about the evaluation itself, and check the scorer before reading anything into the result.
• I5 (answer): Score answers separately on frame or event correctness and on entity grounding, and refuse to state an entity name that no step in the chain from text to world can trace back to the source.