Elamur Research⌕
☰
home / papers / 2022 / Arabic Image Captioning using Pre-training of Deep Bidirectional Transformers

Arabic Image Captioning using Pre-training of Deep Bidirectional Transformers

Paper · 2022 · Jonathan Emami, Pierre Nugues, Ashraf Elnagar et al.

Summary

The paper tackles Arabic image captioning, which lags behind English mainly because there is little public data and because Arabic morphology is complex. The authors take OSCAR, an English vision-language learning method, and replace its text side with pre-trained Arabic and multilingual transformers such as AraBERT. Image region features and detected object tags come from the VinVL visual pipeline. The tags act as anchors that help align image regions with words.

Training uses two resources. Arabic-COCO is a large but noisy Google-translated version of COCO. Arabic Flickr8k, from ElJundi et al., is translated and then checked by people, and it is the only public Arabic benchmark.

The best models beat the earlier published BLEU-1 to BLEU-4 scores on Flickr8k. Two Arabic experts also rated a sample of outputs using the THUMB protocol. The study finds that pairing Arabic captions with English object tags works, but fully Arabic tags would likely be better. It argues that errors in translated labels damage the alignment between images and text. The resulting models give the Arabic captioning community a new baseline.

Key points

  • Best BLEU-1/2/3/4 was 0.39 / 0.25 / 0.15 / 0.092, compared with 0.33 / 0.19 / 0.11 / 0.057 previously. AraBERT32-Flickr8k improves BLEU-4 by about 61.4% over ElJundi et al. (2020).
  • Arabic Flickr8k has a 6,000 / 1,000 / 1,000 image split with three Arabic captions per image, 24,000 captions in total. Arabic-COCO has 414,113 machine-translated captions for 82,783 training images.
  • A sample of 150 Arabic-COCO captions (from Sabri, 2021) found 46% unintelligible. For that reason Arabic-COCO was used only for training and never for validation or testing.
  • Model names follow the pattern modelBatchSize-dataset. SPICE was dropped because its evaluation scripts do not handle Arabic. Human judgments used THUMB, with the closest reference chosen by MUSE similarity.
  • Only 1,114 object-tag translations occur in Flickr8k, plus 253 extra in Arabic-COCO. Some were mistranslated, for example the noun "light" was rendered as an adjective meaning bright.

Interesting ideas

  • I1. Object tags detected in the input serve as explicit anchors that link two representations, which makes alignment easier and speeds up convergence. Symbolic pivots like these can help any cross-representation mapping.
  • I2. The visual encoder and the OSCAR training recipe stay fixed, and only the language model is swapped (AraBERT vs. multilingual) to port the system to a new language. This isolates which component matters.
  • I3. Before using a translated corpus, the authors relied on a measured unintelligibility rate from a small sample, then restricted the noisy data to training and kept evaluation on human-checked data.
  • I4. Errors concentrate in a small, closed set of labels (about 1,100 object tags), so human checking of just that vocabulary is cheap and has high leverage.
  • I5. An automatic metric was dropped once its tooling proved incompatible with the language. Human ratings were then paired with the most semantically similar reference instead of an arbitrary one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): During generation, the entity and type labels in the meaning graph could act as fixed anchors that the realiser must surface, making the graph-to-text alignment checkable.
  • I2 (tree): gparse could keep its parsing head and training loop fixed while trying different pre-trained encoders, so encoder effects are measured cleanly.
  • I3 (learn): Silver data agreed by witness parsers should get a sampled quality audit and should never enter the evaluation sets, which stay gold-only.
  • I4 (world): Mappings from ontology and verb-class labels to lexical items are similarly small and could be human-verified in one pass to stop systematic part-of-speech confusions like noun→adjective.
  • I5 (back): Text → graph → text outputs could be judged against the closest paraphrase reference, and any metric that breaks on certain constructions should be excluded rather than trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Lund UniversityLund · Sweden11 works · 1,263 citations🎓University of SharjahSharjah · United Arab Emirates9 works · 202 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Jonathan Emami, Pierre Nugues, Ashraf Elnagar and Imad Afyouni. 2022. Arabic Image Captioning using Pre-training of Deep Bidirectional Transformers. Proceedings of the 15th International Conference on Natural Language Generation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.inlg-main.4

@inproceedings{emami2022arabic,
  title = {{Arabic Image Captioning using Pre-training of Deep Bidirectional Transformers}},
  author = {Jonathan Emami, Pierre Nugues, Ashraf Elnagar and Imad Afyouni},
  booktitle = {Proceedings of the 15th International Conference on Natural Language Generation},
  year = {2022},
  doi = {10.18653/v1/2022.inlg-main.4},
  url = {https://doi.org/10.18653/v1/2022.inlg-main.4}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aysu Ezen-Can. 2020. A Comparison of LSTM and BERT for Small Corpus. ArXiv, abs/2009.05451. ↗
  2. Imad Afyouni, Imtinan Azhara, and Ashraf Elnagar. 2021. AraCap: A hybrid deep learning architecture for Arabic Image Captioning. In ACLing 2021: 5th International Conference on AI in Computational Linguistics. find
  3. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. ↗
  4. Huda A. Al-muzaini, Tasniem N. Al-yahya, and Hafida Benhidour. 2018. Automatic Arabic image captioning using RNN-LSTM-based language model and CNN. International Journal of Advanced Computer Science and Applications, 9(6). find
  5. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. In 24th International Joint Conference on Artificial Intelligence. ↗
  6. Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021. Scaling Up Vision-Language Pre-training for Image Captioning. CoRR, abs/2111.12233. ↗
  7. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic Propositional Image Caption Evaluation. In Computer Vision – ECCV 2016, pages 382–398, Cham. Springer International Publishing. ↗
  8. Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019. Attention on Attention for Image Captioning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). find
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  10. Vasu Jindal. 2017. A Deep Learning Approach for Arabic Caption Generation Using Roots-Words. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI). find
  11. Vasu Jindal. 2018. Generating image captions in Arabic using root-word based recurrent neural networks and deep neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 144–151, New Orleans, Louisiana, USA. Association for Computational Linguistics.
  12. Anfal Attai and Ashraf Elnagar. 2020. A survey on Arabic Image Captioning Systems Using Deep Learning Models. In 14th International Conference on Innovations in Information Technology (IIT), pages 114–119. find
  13. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  14. Andrej Karpathy and Li Fei-Fei. 2015. Deep Visual-Semantic Alignments for Generating Image Descriptions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↗
  15. Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2021. Transparent Human Evaluation for Image Captioning. CoRR, abs/2111.08940. ↗
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Multilingual bert readme. https://github.com/google-research/bert/blob/master/multilingual.md. [Online; accessed 6 Feb. 2022]. ↗
  17. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics.
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  19. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In Computer Vision – ECCV 2020.
  20. Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, and Daniel Asmar. 2020. Resources and End-to-End Neural Network Models for Arabic Image Captioning. In 15th International Conference on Computer Vision Theory and Applications. find
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing. ↗
  23. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Revisiting Visual Representations in Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5579–5588. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  25. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2019. Unified Vision-Language Pre-Training for Image Captioning and VQA. In Proceedings of the AAAI Conference on Artificial Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/emami-2022-arabic-image-captioning-pre-training/