Elamur Research⌕
☰
home / papers / 2016 / Attention-based Multimodal Neural Machine Translation

Attention-based Multimodal Neural Machine Translation

Summary

This paper, a system for the WMT 2016 multimodal translation shared task, asks whether images can help an attention-based NMT model translate English image captions into German, for instance by disambiguating words. The authors treat visual features as extra encoder steps, so the decoder can attend to image and text jointly. Images are represented by fc7 activations (4096 dimensions) from VGG-19, which a learned matrix maps into the text embedding space. Three encoders are compared:

  • Model 1 adds a single global image vector to the word sequence.
  • Model 2 prepends up to four object regions found by the R-CNN region proposal network, ordered from smallest to largest, plus the whole image.
  • Model 3 runs parallel LSTM threads with shared weights. Each thread starts from one visual feature and then reads the sentence, and the thread outputs are merged by average pooling.

The parallel model worked best. It beat the text-only baseline and was comparable to the strongest Moses-based entries in the challenge. Re-ranking candidates with a bilingual dictionary raised METEOR further. The work is an early demonstration that attending over image regions can aid translation.

Key points

  • Visual input consists of VGG-19 fc7 vectors for the whole image and for the top 4 regions proposed by the Faster R-CNN RPN, with zero-padding when fewer objects are detected. Ordering regions by ascending size beat descending, random and categorical orders.
  • Model 3 (4 regional threads plus 1 global thread) reached 36.5 BLEU and 54.1 METEOR. This is +2.0 BLEU and +2.3 METEOR over the text-only model.
  • Three re-scoring methods were tested on Model 3 candidates:
    • A single-layer LSTM language model (300 hidden units) trained on MSCOCO and IAPR TC-12 captions (56,968 sentences) gave almost no gain.
    • A bilingual autoencoder (256-200-128 layers) lowered both metrics, likely because of overfitting on scarce bilingual data.
    • Bilingual dictionary matching raised METEOR to 55.2 (+1.1), while BLEU dropped to 35.7.
  • The authors suggest adding further modalities, such as audio, as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

57 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 53 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2360

@inproceedings{huang2016attention,
  title = {{Attention-based Multimodal Neural Machine Translation}},
  author = {Huang, Po-Yao and Liu, Frederick and Shiang, Sz-Rung and Oh, Jean and Dyer, Chris},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2360},
  url = {https://doi.org/10.18653/v1/w16-2360}
}

References

10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Galen Andrew, Raman Arora, Karen Livescu, and Jeff Bilmes. 2013. Deep canonical correlation analysis. In International Conference on Machine Learning (ICML), Atlanta, Georgia. find
  2. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67– 78. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  4. Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually-constrained phrase embeddings for machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 111–121, Baltimore, Maryland, June. Assoc… find
  5. D. Elliott, S. Frank, K. Sima’an, and L. Specia. 2016. Multi30K: Multilingual English-German Image Descriptions. CoRR, abs/1605.00459. find our card
  6. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition. find our card
  7. David R. Hardoon, Sandor R. Szedmak, and John R. Shawe-taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural Comput., 16(12):2639–2664, December. find
  8. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. CoRR, abs/1412.2007. find our card
  9. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. Seattle, October. Association for Computational Linguistics. find
  10. Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/huang-2016-attention-based-multimodal-neural-machine/