Elamur Research⌕
☰
home / papers / 2017 / Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Paper · 2017 · Iacer Calixto, Qun Liu and Nick Campbell

Summary

The paper asks whether images can measurably improve neural machine translation of image captions. Earlier multi-modal systems that used spatial image features rarely beat text-only or global-feature baselines. The authors extend the attention-based encoder–decoder of Bahdanau et al. into a model called NMT SRC+IMG. Its decoder, a "doubly-conditional" GRU, has two separate attention mechanisms: one over source-word annotations and one over image regions. The image regions come from a pre-trained ResNet-50. A learned gate scales how much visual context each generated word receives. The model is trained on English→German data from the translated part of Multi30k (M30k T). It is additionally pre-trained on back-translated in-domain multi-modal data and on large general-domain text-only MT corpora, and it benefits from both. It sets a new state of the art on the M30k T test set and compares favourably with attention-based NMT and PB-SMT baselines trained on the same data. The work shows that visual attention can help translation when sentences describe objects visible in the image.

Key points

  • Visual features are taken from the res4f layer of ResNet-50: a 14×14 grid flattened into 196 region vectors of 1024 dimensions each.
  • M30k T pairs each Flickr30k image with one English caption and its professional German translation. The splits are 29k images for training, 1,014 for validation and 1k for test.
  • The gating scalar β is above 0.5 for about 20% of output words and above 0.8 for only 8%. These words are mostly concrete, visually salient nouns.
  • The model has about 213M parameters versus about 200M for the text-only attention baseline, roughly 6.6% more.
  • Comparison baselines include the models of Huang et al. (2016), which used VGG19 global features and RCNN region features. The authors describe these as the strongest previously published purely multi-modal NMT results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dublin City UniversityDublin · Ireland72 works · 9,746 citations🎓Trinity College DublinDublin · Ireland25 works · 1,313 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,105 worksVision & language · 811 works

Cite

Cite as Iacer Calixto, Qun Liu and Nick Campbell. 2017. Doubly-Attentive Decoder for Multi-modal Neural Machine Translation. arXiv:1702.01287.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1702.01287 · arXiv 1702.01287

@misc{calixto2017doubly,
  title = {{Doubly-Attentive Decoder for Multi-modal Neural Machine Translation}},
  author = {Iacer Calixto, Qun Liu and Nick Campbell},
  year = {2017},
  doi = {10.48550/arxiv.1702.01287},
  eprint = {1702.01287},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1702.01287}
}

References

44 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, ICLR 2015. San Diego, California.
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. J. Mach. Learn. Res. 3:1137–1155. http://dl.acm.org/citation.cfm?id=944919.944966. ↗
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc…
  4. Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, and Joost van de Weijer. 2016. Does multimodality help human and machine for translation and image captioning? In Proceedings of the First Conference on Machine Translation. Berlin, Ger… find
  5. Iacer Calixto, Desmond Elliott, and Stella Frank. 2016. Dcu-uva multimodal mt system report. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, pages 634–638. http://www.aclweb.org/anthology/W/W16/W16-2359. ↗
  6. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Translation. page 103.
  7. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using rnn encoder– decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lang…
  8. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Pap… find
  9. Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
  10. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-Task Learning for Multiple Language Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon…
  11. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-Language Image Description with Neural Sequence Models. CoRR abs/1510.04709. http://arxiv.org/abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) ↗
  12. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German Image Descriptions. In Proceedings of the 5th Workshop on Vision and Language, VL@ACL 2016. Berlin, Germany. http://aclweb.org/anthology/W/W16/W16-3210.pdf. ↗
  13. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, Cal…
  14. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems, NIPS, Barcelona, Spain, pages 1019–1027. http://papers.nips.cc/paper/6241-a-theoretically-grounded-application-of-dropout-in-recu… ↗
  15. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC, USA, CVPR ’14, pages 580–587. https://doi.or… ↗
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 .
  17. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal Pivots for Image Caption Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 2399–2409. http://www.aclweb.org/anthology/P16-1… ↗
  18. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based Multimodal Neural Machine Translation. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, pages 639–645. http://www.aclweb.org/anthology/W/W16/W16-2360. ↗
  19. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P…
  20. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013. Seattle, US., pages 1700–1709.
  21. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Detroit, Michigan, volume I, pages 181–184. find
  22. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit. AAMT, AAMT, Phuket, Thailand, pages 79–86. http://mt-archive.info/MTS-2005-Koehn.pdf. ↗
  23. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Pro… find
  24. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical Phrase-based Translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1. Edmonton, Canada, NAACL ’03, pages 48– 54. https… find
  25. Jindřich Libovický, Jindřich Helcl, Marek Tlustý, Ondřej Bojar, and Pavel Pecina. 2016. CUNI System for WMT16 Automatic Post-Editing and Multimodal Translation Tasks. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, pages 646–654. http://www.aclweb.org/anthology/W… ↗
  26. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-Task Sequence to Sequence Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. San Juan, Puerto Rico.
  27. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Lisbon, Portugal, pages 1412–1421.
  28. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. 2014. Explain Images with Multimodal Recurrent Neural Networks. http://arxiv.org/abs/1410.1090. ↗
  29. Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1. Sapporo, Japan, ACL ’03, pages 160–167. https://doi.org/10.3115/1075096.1075117. ↗
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Philadelphia, Pennsylvania, ACL ’02, pages 311–318. https://doi.org/10.3115/10730… ↗
  31. Maja Popović. 2015. chrf: character n-gram fscore for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Lisbon, Portugal, pages 392–395. http://aclweb.org/anthology/W15-3049. ↗
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 86–96. http://www.aclweb.org/antholog… ↗
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 1715–1725. http://www.aclweb.org/anthology/… ↗
  34. Kashif Shah, Josiah Wang, and Lucia Specia. 2016. SHEF-Multimodal: Grounding Machine Translation on Images. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, pages 660–665. http://www.aclweb.org/anthology/W/W16/W16-2363. ↗
  35. K. Simonyan and A. Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR). San Diego, CA.
  36. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas. Cambridge, MA, pages 223–231.
  37. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A Shared Task on Multimodal Machine Translation and Crosslingual Image Description. In Proceedings of the First Conference on Machine Translation, WMT 2016, colocated with ACL 2016. Berlin, Germany, pages 543– 553. http://aclweb.… ↗
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems. Montréal, Canada, pages 3104–3112.
  39. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling Coverage for Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 76–85. http://www.aclweb.org/anthology/P16… ↗
  40. Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2015. Sequence to sequence - video to text. In 2015 IEEE International Conference on Computer Vision, ICCV 2015. Santiago, Chile, pages 4534–4542. https://doi.org/10.1109/ICCV.2015.515. ↗
  41. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015. Boston, Massachusetts, pages 3156–3164.
  42. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15). JMLR Workshop an…
  43. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2:67–78. find
  44. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR abs/1212.5701. http://arxiv.org/abs/1212.5701. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/calixto-2017-doubly-attentive-decoder-multi-modal/