Elamur Research⌕
☰
home / papers / 2020 / Multimodal Transformer for Multimodal Machine Translation

Multimodal Transformer for Multimodal Machine Translation

Paper · 2020 · Shaowei Yao and Xiaojun Wan

Summary

The paper addresses multimodal machine translation (MMT), where an image accompanies the source sentence. The authors argue that text should dominate because raw image features carry much irrelevant content, and treating both inputs equally can add noise. They propose a Transformer encoder with multimodal self-attention. Following a graph view of Transformer, image regions are added as extra pseudo-word nodes next to the words. However, the image nodes' hidden states are built only from the word representations. The visual features serve solely to steer how the attention weights are assigned, so the image is never encoded directly. Experiments use Multi30k English→German (Test2016), with ResNet-50 spatial features, optionally adding back-translated training data. The model beats text-only and earlier multimodal systems on BLEU and reaches a new state of the art once extra data is included. Ablations show the gains come from relevant visual input. The design offers a simple way to let a weaker modality support a stronger one, which may transfer to other unbalanced multimodal tasks.

Key points

  • Data: Multi30k (M30k_T for training; 145k German captions from M30k_C back-translated with an LSTM De→En model). Features are 7×7×2048 grids from ResNet-50's final convolutional layer. Setup uses GloVe 300-d embeddings and 6 encoder plus 6 decoder layers with 10 heads.
  • On original data, the model scores 38.7 BLEU4 and 55.7 METEOR. This beats the text-only Transformer (37.8/55.3) and the best prior BLEU (Del+obj, 38.0), while VMMT_F keeps a slightly higher METEOR (56.0).
  • With back-translated data, the model reaches 39.5 BLEU4 and 56.9 METEOR, surpassing VMMT_F (38.4/56.3) and IMG_D (38.5/55.9).
  • Ablation: swapping multimodal self-attention for plain self-attention over concatenated word and image features costs 0.6 BLEU and 0.4 METEOR. Blank images reduce scores to 37.1/54.8, and random images reduce them to 36.7/54.5, which is below the text-only baseline.
  • Attention visualizations indicate that heads focus on caption-relevant image regions, such as a building or a balance beam, which helps resolve translation ambiguities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China100 works · 35,618 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 958 worksVision & language · 269 works

Cite

Cite as Shaowei Yao and Xiaojun Wan. 2020. Multimodal Transformer for Multimodal Machine Translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.400

@inproceedings{yao2020multimodal,
  title = {{Multimodal Transformer for Multimodal Machine Translation}},
  author = {Shaowei Yao and Xiaojun Wan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.400},
  url = {https://doi.org/10.18653/v1/2020.acl-main.400}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loı̈c Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304–323.
  2. Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.
  3. Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In EMNLP, pages 992–1003. find
  4. Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Doubly-attentive decoder for multi-modal neural machine translation. In ACL, pages 1913–1924. find
  5. Desmond Elliott and Ákos Kádár. 2017. Imagination improves multimodal translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 130–141. find
  6. Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019. Distilling translations with visual awareness. In ACL, pages 6525–6538. find
  7. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR. find
  8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311– 318.
  9. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL, pages 86–96. find
  10. Lucia Specia, Stella Frank, Khalil Simaan, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 543–553.
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008.
  12. Shaowei Yao, Tianming Wang, and Xiaojun Wan. 2020. Heterogeneous graph transformer for graph-to-sequence learning. In Proceedings of the Association for Computational Linguistics (ACL). find
  13. Iacer Calixto, Miguel Rios, and Wilker Aziz. 2019. Latent variable model for multi-modal translation. In ACL, pages 6392–6405. find
  14. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380.
  15. Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In EMNLP, pages 2974–2978. find
  16. Desmond Elliott, Stella Frank, Loı̈c Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215–233.
  17. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2020-multimodal-transformer-multimodal-machine-translation/