📄 AraCLIP: Cross-Lingual Learning for Effective Arabic Image Retrieval

Summary

AraCLIP targets text-to-image retrieval with Arabic queries. Most CLIP-style systems are English-centric, and multilingual alternatives handle Arabic's morphology and cultural vocabulary only loosely. The authors do not retrain CLIP contrastively. Instead they use Knowledge Distillation: the English CLIP text encoder is frozen as a teacher, and a pre-trained AraBERT student learns to output the same embedding for the Arabic machine translation of each English caption. Because the English encoder already lives in the same space as the CLIP image encoder, the Arabic student inherits that alignment, and training never touches an image. The training data are machine-translated and cleaned Arabic versions of CC3M+CC12M+SBU and MS COCO. Evaluation uses Arabic XTD-11 and Arabic Flicker 8k. Against the multilingual mCLIP baseline, AraCLIP gains roughly 10% on MRR and Recall, and the best variants reach MRR@1 above 0.66 on XTD-11. The work matters because it shows a cheap, image-free recipe for bringing a strong English vision-language model to a new language. The datasets, code and a Hugging Face demo are public.

Key points

• Released resources:
  • about 12.5M Arabic caption-image pairs, translated from CC3M+CC12M+SBU (synthetic captions filtered by ViT-L);
  • about 123,280 translated MS COCO pairs;
  • an Arabic XTD-11 test set of about 1,000 pairs.
• Training setup:
  • The teacher is a frozen English CLIP text encoder; the student is Arabert-v2 (base and large).
  • The loss is MSE between embeddings, chosen over cosine similarity.
  • The image towers are ViT-B-16-plus-240 and ViT-B-16-SigLIP-512.
• Data splits: a 2M training subset with 5,000 validation pairs, optionally plus 113,281 MS COCO training pairs. Evaluation uses XTD-11 (1,000 pairs) and Arabic Flicker 8k (8,000 images).
• Results by variant:
  • Arabert-v2-base-ViT-B-16-SigLIP-512-2M, with or without MS COCO, is best and beats mCLIP (XLM-Roberta-Large-Vit-B-16Plus) on MRR@1/5/10 and R@1/5/10.
  • Large students do not beat the base ones.
  • Adding MS COCO helps slightly, especially at R@5 and R@10.
• A qualitative study probes two categories with top-3 retrievals: compositional sentences (e.g., a windsurfing query) and noisy cases where the mentioned objects are hard to see.

Interesting ideas

• I1. The student never sees an image; it only has to land on the frozen teacher's text embedding, and cross-modal alignment comes along for free, which makes the method far cheaper than contrastive training.
• I2. The authors regress embeddings with MSE rather than maximizing cosine similarity, citing earlier work that MSE gives a richer training signal for distillation.
• I3. Machine-translated captions are post-processed and cleaned before training, so translation output is treated as silver data that still needs filtering.
• I4. The base student outperformed the large one, which shows that more capacity did not pay off for this task.
• I5. The qualitative tests are grouped by phenomenon (compositional sentences versus partially visible objects) instead of relying only on aggregate scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): A non-English sentence encoder could be distilled onto Mova's English meaning representations using translated parallel text, so that translation through the meta-world reuses the English pipeline instead of building a separate one.
• I2 (tree): When distilling gparse into a lighter or faster parser, the student's hidden states could be regressed onto the teacher's with MSE, rather than matching only the final arc and label decisions.
• I3 (learn): Any silver corpus produced by a translation or LLM step should pass an explicit cleaning gate before it reaches agreement among witness parsers or training.
• I4 (learn): Small and large variants should be compared on the same held-out sets before committing to bigger models, with the scaling result kept as an evaluation record.
• I5 (answer): School-test and fable-QA evaluations could be tagged by phenomenon (negation, embedded clauses, implicit entities) so that failures point to a specific stage of the pipeline.