Elamur Research⌕
☰
home / papers / 2018 / Generating Image Captions in Arabic using Root-Word Based Recurrent Neural Networks and Deep Neural Networks

Generating Image Captions in Arabic using Root-Word Based Recurrent Neural Networks and Deep Neural Networks

Paper · 2018 · Vasu Jindal

Summary

The paper tackles image captioning directly in Arabic. Earlier work was almost entirely English, and machine-translating English captions loses detail because Arabic morphology is rich. The author exploits the Semitic root system in a three-stage pipeline:

  • An RCNN detector maps image regions onto Arabic root-words instead of English words.
  • A root-word based recurrent network (rwRNN, LSTM cells) expands those roots into full Modern Standard Arabic words and a sentence.
  • Dependency tree constraints from the Prague Arabic Dependency Treebank (PADT) check whether the word order is grammatical.

Evaluation uses two datasets: Flickr8k with Arabic captions written by professional translators, and about 405,000 captioned images from bilingual Middle Eastern newspaper sites. The baselines are standard English captioners whose output is passed through Google Translate. The direct root-based system beats them on BLEU and METEOR. The paper claims these are the first reported BLEU scores for Arabic caption generation. Its broader argument is that generating natively in a morphologically rich language, around its core lexical units, beats translating from a pivot language.

Key points

  • Vision stage: the RCNN is pre-trained on ImageNet and fine-tuned on the 200 ImageNet Detection classes. It uses the top 19 detected regions plus the whole image. During training, English object labels are paired with Arabic root labels.
  • Root extraction: Arabic root labels come from a transducer-based root extractor (Yaseen and Hmeidi; Yousef et al.). It combines noun/verb, pattern and suffix transducers, drawing on a set of 4 prefixes for verbs, 12 prefixes for nouns and 20 frequently occurring suffixes.
  • rwRNN input: each root is encoded as three positional letter slots. For four- or five-letter roots, the middle slot absorbs all the inner letters.
  • Stage 3 objective: the score adds content selection and linguistic fluency, subject to hard dependency constraints (after Kuznetsova et al., 2013) built on PADT.
  • Results:
DatasetMetricThis paperBest baseline (Visual Attention + Google Translate)
Flickr8kBLEU-165.854.2
Flickr8kBLEU-422.319.4
Flickr8kMETEOR20.09not reported
NewsBLEU-155.648.5
NewsBLEU-418.915.4
NewsMETEOR18.0116.82

The main remaining errors involve recent words that are not built from roots.

Interesting ideas

  • I1. Representing a word by fixed positional slots for its core skeleton, with one elastic slot for irregular length, gives the network a compact input that is shared across a whole derivational family.
  • I2. Predicting an abstract lexical unit first and realising the surface word only afterwards splits "what to say" from "how it inflects".
  • I3. Candidate sentences are scored on content fit plus fluency but must also satisfy hard treebank-derived dependency constraints, so ungrammatical outputs are filtered out rather than merely penalised.
  • I4. The evaluation pits direct target-language generation against a "generate in English, then machine-translate" pivot, which shows how much the pivot step costs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A similar slotted encoding of English derivational families (e.g., decide/decision/decisive sharing a base) could feed the tagger and lemmatizer as an explainable feature instead of opaque subword pieces.
  • I2 (back): Generation from the meaning graph could first choose lemmas and frames, then run a separate, inspectable morphology-realisation step.
  • I3 (back): The text → graph → text compressor could reject any regenerated sentence whose UD parse violates hard constraints, with the violated rule logged as the explanation.
  • I4 (back): Translation through the meta-world should be benchmarked against a plain MT pivot baseline on the same references, to show whether the meaning detour actually adds value.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States61 works · 5,614 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Vasu Jindal. 2018. Generating Image Captions in Arabic using Root-Word Based Recurrent Neural Networks and Deep Neural Networks. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-4020

@inproceedings{jindal2018generating,
  title = {{Generating Image Captions in Arabic using Root-Word Based Recurrent Neural Networks and Deep Neural Networks}},
  author = {Vasu Jindal},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop},
  year = {2018},
  doi = {10.18653/v1/n18-4020},
  url = {https://doi.org/10.18653/v1/n18-4020}
}

References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137. ↗
  2. Thamir Al Barrag. 2014. Noun phrases in urban hijazi arabic: A distributed morphology approach. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗
  4. Andrej Karpathy, Armand Joulin, and Fei Fei F Li. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in neural information processing systems, pages 1889–1897. ↗
  5. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. ↗
  6. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105.
  7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE. find
  8. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2011. Baby talk: Understanding and generating image descriptions. In Proceedings of the 24th CVPR. find
  9. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1473–1482. find
  10. Charles A Ferguson. 1959. 15(2):325–340. Diglossia. find
  11. Polina Kuznetsova, Vicente Ordonez, Alexander C Berg, Tamara L Berg, and Yejin Choi. 2013. Generalizing image captions for image-text parallel corpus. In ACL (2), pages 790–796. word, Yuval Marton, Nizar Habash, and Owen Rambow. 2010. Improving arabic dependency parsing with lexical and inflectional morphological features. In Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 13–21. Association for Computational Linguistics.
  12. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587.
  13. Takashi Miyazaki and Nobuyuki Shimizu. 2016. Cross-lingual image caption generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1780–1790. find
  14. Jan Hajic, Otakar Smrz, Petr Zemánek, Jan Šnaidauf, and Emanuel Beška. 2004. Prague arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools, pages 110–117. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. ↗
  16. Yannis Haralambous, Yassir Elidrissi, and Philippe Lenca. 2014. Arabic language text classification using dependency syntax-based feature selection. arXiv preprint arXiv:1410.4863. ↗
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  18. Hao Peng and Nianhen Li. 2016. Generating chinese captions for flickr30k images. find
  19. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675–678. ACM.
  20. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252. ↗
  21. Vasu Jindal. 2017. A deep learning approach for arabic caption generation using roots-words. In AAAI, pages 4941–4942. find
  22. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 806–813. ↗
  23. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, 39, page 413.
  24. Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218. ↗
  25. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575. ↗
  26. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164. ↗
  27. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, pages 77–81. ↗
  28. Qussai Yaseen and Ismail Hmeidi. 2014. Extracting the roots of arabic words without removing affixes. Journal of Information Science, 40(3):376–385. ↗
  29. Nidal Yousef, Aymen Abu-Errub, Ashraf Odeh, and Hayel Khafajeh. 2014. An improved arabic word’s roots extraction method using n-gram technique. Journal of Computer Science, 10(4):716. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jindal-2018-generating-image-captions-arabic-root/