Elamur Research⌕
☰
home / papers / 2025 / AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks

AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks

Paper · 2025 · Mohamed Eltahir, Osamah Sarraj, Abdulrahman M. Alfrihidi et al.

Summary

Video-text retrieval research is evaluated almost entirely on English benchmarks, with some Chinese and European-language coverage, which leaves Arabic without a public test set. The authors propose AutoArabic, a three-step pipeline for localizing an existing benchmark into Modern Standard Arabic. First, an LLM (Gemini 2.0 Flash) translates each caption. Second, a different LLM (GPT-4o) reviews each output and labels likely problems using six error categories. Third, native speakers correct the flagged items. Applied to DiDeMo, the pipeline produces DiDeMo-AR, which keeps the original 10,464 Flickr videos and 26,892 moments and adds 40,144 Arabic captions. The automatic checker agrees with human reviewers at 97% accuracy, and the authors estimate this cuts manual revision roughly fourfold. A lightweight CLIP-style model trained identically on both language versions loses less than 3 points of Recall@1 on Arabic. This suggests the localized benchmark is about as hard as the original. The paper also contributes an error taxonomy and open code, giving a reusable recipe for localizing benchmarks into other languages.

Key points

  • DiDeMo-AR keeps DiDeMo's train/val/test splits (8,395 / 1,065 / 1,004 videos), and its Arabic captions are shorter than the English ones: 5.6 versus 7.5 words on average.
  • The checker uses six tags: lexical, literal, hallucination, tense_shift, loanword and diacritics.
    • It reaches 97% accuracy and a macro F1 of 0.91.
    • Diacritics are detected perfectly.
    • Tense shifting is the hardest class (F1 0.80).
  • The baseline setup:
    • Frozen ViT-B/16 or ViT-B/32 vision towers with a trainable 256-d projection head.
    • paraphrase-multilingual-mpnet-base-v2 as the text encoder and a symmetric InfoNCE loss.
    • 8 frames per video and six epochs on one A100.
  • Results of the baseline:
    • The Arabic drop in R@1 is under 3 pp in both retrieval directions.
    • Arabic keeps about 85–90% of the English R@10.
    • Median rank rises by 3–4 positions.
  • Three post-editing budgets were compared: raw output (zero), fixing only flagged captions (few), and fixing everything (full).
    • Full correction adds roughly 2 pp R@1 over raw output, and raw output is still usable.
    • The gains are clear for ViT-B/16 but nearly flat for ViT-B/32.
  • Five annotators actually revised every caption, which is what made it possible to measure the checker's quality against human judgment.

Interesting ideas

  • I1. Generation and auditing are split between two different LLMs, and the auditor works from a fixed, named error typology, so humans only see items it flags
  • I2. Although selective review was the goal, the authors reviewed everything once so they could measure the filter's per-class precision and recall
  • I3. The downstream task is evaluated under a ladder of correction budgets (none, flagged-only, all), which shows what each unit of human effort actually buys
  • I4. Diacritics were left unconstrained during generation and made uniform afterwards by deterministic stripping, treating inconsistency rather than presence as the error
  • I5. Benchmark difficulty is checked by running one fixed model with identical settings on the original and the translated version, so any gap reflects the language change

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A separate critic model could label silver parses or LLM-teacher outputs with specific failure types (wrong head, wrong relation, wrong lemma). Only those flagged sentences would then go to a stronger teacher tier or a human.
  • I2 (learn): Periodically annotating a full sample would show how much the witness-agreement filters and self-induced repair rules miss. This acts as a negative control for the filtering step itself.
  • I3 (tree): gparse could be trained on raw silver, flag-corrected and fully corrected treebank versions. Comparing LAS across these versions would tell us where annotation effort pays off.
  • I4 (tokens): Spelling variants, quote styles and hyphenation could be normalized by a deterministic pass after tokenization. This would replace trying to force upstream sources into one convention.
  • I5 (back): The same reader could answer QA on an original fable and on its text → graph → text reconstruction. The score gap would measure how much meaning the round trip loses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Mohamed Eltahir, Osamah Sarraj, Abdulrahman M. Alfrihidi, Taha Alshatiri, Mohammed Khurd, Mohammed Bremoo and Tanveer Hussain. 2025. AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.23

@inproceedings{eltahir2025autoarabic,
  title = {{AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks}},
  author = {Mohamed Eltahir, Osamah Sarraj, Abdulrahman M. Alfrihidi, Taha Alshatiri, Mohammed Khurd, Mohammed Bremoo and Tanveer Hussain},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.23},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.23}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabicaqa: A comprehensive dataset for arabic question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2049–2059.
  2. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2023. Masc: Massive arabic speech corpus. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1006–1013. IEEE. find
  3. Ahmed Ali, Peter Bell, James R. Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279–284, San Diego, CA, USA. IEEE.
  4. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic mgb-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316–322. IEEE.
  5. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803–5812. find
  6. Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970. find
  7. Qifeng Cai, Hao Liang, Hejun Dong, Meiyi Qiang, Ruichuan An, Zhaoyang Han, Zhengzhou Zhu, Bin Cui, and Wentao Zhang. 2025. Lovr: A benchmark for long video retrieval in multimodal contexts. arXiv:2505.13928. ↗
  8. Google Cloud. 2025. Gemini 2.0 flash on vertex ai: Low-latency multimodal generation. https://cloud.google.com/vertex-ai/docs/ generative-ai/model-reference/gemini. Accessed June 2025. ↗
  9. Rishabh Dabral, Ganesh Ramakrishnan, Preethi Jyothi, and 1 others. 2021. Rudder: A cross lingual video and text retrieval dataset. arXiv preprint arXiv:2103.05457. ↗
  10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. ↗
  11. Salah Elhaj and Waleed Abdulla. 2021. Avsd-arabic: An audio-visual lip-reading dataset for modern standard arabic. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). find
  12. Amal Elnagar and Ahmed Gouza. 2020. Anad: Arabic natural audio dataset for speech emotion recognition. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find
  13. Meghan Glenn, Haejoong Lee, Stephanie Strassel, and Kazuaki Maeda. 2017. Gale phase 4 arabic broadcast conversation transcripts. LDC2017T12, Web Download. ↗
  14. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University - Computer and Information Sciences, 33(5):497–507.
  15. Abdelhamid Haouhat, Slimane Bellaouar, Attia Nehar, and Hadda Cherroun. 2023. Towards arabic multimodal dataset for sentiment analysis. In 2023 Fourth International Conference on Intelligent Data Science Technologies and Applications (IDSTA), pages 126–133. IEEE.
  16. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591. find
  17. Willy Fitra Hendria. 2023. Msvd-indonesian: A benchmark for multimodal video-text tasks in indonesian. arXiv preprint arXiv:2306.11341. ↗
  18. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296. find
  19. OpenAI. 2025. GPT-4o: Openais omnimodal flagship model. https://openai.com/blog/ gpt-4o. Accessed June 2025. ↗
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR.
  21. Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A dataset for movie description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202–3212. ↗
  22. Jack Shepherd. 2025. 23 essential youtube statistics you need to know in 2025. https://thesocialshepherd.com/blog/ youtube-statistics. Updated June 11, 2025; accessed July 2, 2025. ↗
  23. Francisco Soldan and 1 others. 2022. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  24. Kaitao Song, Xu Tan, Tianyang Zhang, Rui Wang, Liang Lu, Ada Lin, Qingyu Zhou, Lu Zhang, and Furu Wei. 2020. MPNet: Masked and Permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297. ↗
  25. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:1807.03748. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eltahir-2025-autoarabic-three-stage-framework-localizing/