Elamur Research⌕
☰
home / papers / 2021 / Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Paper · 2021 · Zhiyong Wu, Lingpeng Kong, Wei Bi et al.

Summary

The paper asks whether multimodal machine translation (MMT) systems really benefit from images, or whether their reported gains come from something else. Earlier work mostly relied on adversarial comparisons, such as feeding random images, which give only indirect evidence. The authors instead build two interpretable models:

  • Gated Fusion, for conventional MMT with a paired image.
  • RMMT, a retrieval-augmented variant that fetches the top-K relevant images using a BERT/ResNet dense retriever.

In both models, a learnable gating matrix Λ shows how much visual signal is mixed into the Transformer encoder's textual representation.

The models were evaluated on Multi30k and the video-based VaTex. They match the improvements of recent multimodal systems, yet the gate values show that they largely ignore visual input. Further analysis links the gains to a regularization effect, similar to weight decay or random noise injection. Images help only when textual context is deliberately degraded, for example by masking visually grounded tokens.

The findings call for new benchmarks where vision is truly necessary. They also position interpretable components as a verification tool for future MMT research.

Key points

  • Data. Multi30k uses a split of 29,000 / 1,014 / 1,000 instances (Test2016), plus Test2017 (1,000) and an out-of-domain MSCOCO test set (461 instances with ambiguous verbs). Experiments cover En-De and En-Fr, with BPE at 10,000 merges, giving vocabularies of 9,712 and 9,544 tokens respectively.
  • Features and retriever. Visual features are 2048-dimensional average-pooled ResNet-50 vectors. The RMMT retriever pairs bert-base-uncased (CLS pooling) with inner-product similarity. It is pre-trained on Flickr30k with Multi30k overlaps removed and evaluated by R@K.
  • Masked-input results. When visually grounded tokens (more than 30 occurrences, stop words excluded) are masked, ResNet-based models clearly outperform text-only ones. RMMT reaches nearly 50% higher BLEU than the Transformer, while weight decay and noise injection give only marginal gains.
  • Remaining limits. Even with images, translation quality under masking stays well below full-text levels. Abstract concepts cannot be visualized, and attention to the wrong image regions causes errors.
  • Code. Code and baselines are available at github.com/LividWo/Revisit-MMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China66 works · 17,945 citations🎓University of Hong KongPok Fu Lam · China44 works · 10,936 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China44 works · 10,162 citations🎓East China Normal UniversityShanghai · China19 works · 1,856 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 958 worksVision & language · 269 worksInformation retrieval & search · 145 works

Cite

Cite as Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li and Ben Kao. 2021. Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.480

@inproceedings{wu2021good,
  title = {{Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation}},
  author = {Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li and Ben Kao},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.480},
  url = {https://doi.org/10.18653/v1/2021.acl-long.480}
}

References

52 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guozhong An. 1996. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643–674. find
  2. Hasan Sait Arslan, Mark Fishel, and Gholamreza Anbarjafari. 2018. Doubly attentive transformer machine translation. arXiv preprint arXiv:1807.11605. — [arXiv:1807.11605](https://arxiv.org/abs/1807.11605) find
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  4. Loı̈c Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In WMT18.
  5. Chris M Bishop. 1995. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116. find
  6. Christopher M Bishop et al. 1995. Neural networks for pattern recognition. Oxford university press. find
  7. Ozan Caglayan, Julia Ive, Veneta Haralampieva, Pranava Madhyastha, Loı̈c Barrault, and Lucia Specia. 2020. Simultaneous machine translation with visual context. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2350–2361, Online. Association for… find
  8. Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loı̈c Barrault. 2019. Probing the need for visual context in multimodal machine translation. In NAACL-HLT (1).
  9. Iacer Calixto, Desmond Elliott, and Stella Frank. 2016. Dcu-uva multimodal mt system report. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 634–638. find
  10. Iacer Calixto and Qun Liu. 2017. Sentence-level multilingual multi-modal embedding for natural language processing. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 139–148, Varna, Bulgaria. INCOMA Ltd. find
  11. Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Doubly-attentive decoder for multi-modal neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1913– 1924, Vancouver, Canada. Association for Computational… find
  12. Koel Dutta Chowdhury and Desmond Elliott. 2019. Understanding the effect of textual adversaries in multimodal machine translation. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN), pages 35–40. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… find
  15. Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2974–2978. find
  16. Desmond Elliott, Stella Frank, Loı̈c Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215–233.
  17. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multilingual image description with neural sequence models. arXiv preprint arXiv:1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) find
  18. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74.
  19. Desmond Elliott and Ákos Kádár. 2017. Imagination improves multimodal translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 130–141. find
  20. Stella Frank, Desmond Elliott, and Lucia Specia. 2018. Assessing multilingual multimodal image description: Studies of native speaker preferences and translator choices. Natural Language Engineering, 24(3):393–413. find
  21. Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018a. The MeMAD submission to the WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Tra… find
  22. Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, et al. 2018b. The memad submission to the wmt18 multimodal translation task. arXiv. find
  23. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In AAAI, pages 5133–5140. find
  24. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. ICML. find
  25. Stephen José Hanson and Lorien Y Pratt. 1989. Comparing biases for minimal network construction with back-propagation. In Advances in neural information processing systems, pages 177–185. find
  26. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. find
  27. Jindřich Helcl, Jindřich Libovickỳ, and Dušan Variš. 2018. Cuni system for the wmt18 multimodal translation task. arXiv preprint arXiv:1811.04697. find
  28. Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? it’s harder to tell than you might think! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 861–877. find
  29. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based multimodal neural machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 639–645, Berlin, Germany. Association for Computational Linguistics.
  30. Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019. Distilling translations with visual awareness. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6525– 6538, Florence, Italy. Association for Computational Linguistics. find
  31. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739. Springer. find
  32. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  33. Jan Kukačka, Vladimir Golkov, and Daniel Cremers. 2017. Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710.10686. — [arXiv:1710.10686](https://arxiv.org/abs/1710.10686) find
  34. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. 34th Conference on Neural Information Processing Systems (NeurI… find
  35. Jindřich Libovický and Jindřich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196–202, Vancouver, Canada. Association for Computational Li…
  36. Jindřich Libovický, Jindřich Helcl, and David Mareček. 2018. Input combination strategies for multi-source transformer decoder. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 253–260, Belgium, Brussels. Association for Computational Linguistics. find
  37. Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020. Dynamic context-guided capsule network for multimodal machine translation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1320–1329. find
  38. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
  39. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649… find
  40. Justin Salamon and Juan Pablo Bello. 2017. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24(3):279–283. find
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
  42. Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu. 2018. A visual attention grounding neural model for multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3643– 3653. find
  43. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 543–553.
  44. Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. 2019. Incorporating bert into neural machine translation. In International Conference on Learning Representations. find
  45. Hao Tan and Mohit Bansal. 2020. Vokenization: Improving language understanding via contextualized, visually-grounded supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2066– 2080. find
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  47. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In The IEEE International Conference on Computer Vision (ICCV). find
  48. Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87–92.
  49. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2018. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
  50. Shaowei Yao and Xiaojun Wan. 2020. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4346– 4350.
  51. Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020. A novel graph-based multi-modal fusion encoder for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025–3035. find
  52. Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal visual representation. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2021-good-misconceived-reasons-empirical-revisiting/