Elamur Research⌕
☰
home / papers / 2019 / Probing the Need for Visual Context in Multimodal Machine Translation

Probing the Need for Visual Context in Multimodal Machine Translation

Summary

Earlier multimodal machine translation (MMT) studies found that images add little or nothing. Many attributed this to weak image features or poor fusion methods. The authors propose another cause: the Multi30K captions are short, simple and repetitive, so the English source already holds enough context and the image becomes redundant.

To test this, they deliberately remove information from the source text in both training and test data. They mask colour words, mask head nouns of visually grounded entities, and keep only the first k tokens. They then compare a text-only attentive GRU NMT baseline with three MMT variants (DIRECT, HIER, INIT) that use ResNet-50 image features. They also measure visual sensitivity by pairing test sentences with mismatched images.

With complete sentences, the multimodal systems are not significantly better than the baseline. Once textual context is reduced, they clearly outperform NMT and are hurt by mismatched images, so they do use the visual input. The work suggests that MMT needs data where image and text are complementary. It also suggests that visual grounding could make translation more robust to noisy input.

Key points

  • Setup: Main experiments use English→French Multi30K (about 30K training sentences, evaluated on test2017). Entity masking is also run for German and Czech using Flickr30K entity annotations. Scores are METEOR averaged over three runs, and the models are built with nmtpytorch.
  • Full data: On unmodified data, NMT scores 70.6 METEOR and the MMT systems reach 70.7–70.9, which is not a significant difference.
  • Colour deprivation: All MMT models become significantly better than NMT. On the 247 affected test sentences, HIER gains 1.6 METEOR. Colour accuracy rises from 32.5% to 44.5% with the attentive models, compared with 36.5% for INIT.
  • Entity masking: Attentive models gain up to 4.2 METEOR on French, with smaller gains for German and Czech. Mismatched images cut scores by up to 9.7 points.
  • Progressive masking: At k=0 the task effectively becomes image captioning, and MMT beats NMT by about 7 METEOR. Sensitivity to mismatched images shrinks as more source words remain. Models trained on mismatched images from the start ("blinded") perform no better than NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1422

@inproceedings{caglayan2019probing,
  title = {{Probing the Need for Visual Context in Multimodal Machine Translation}},
  author = {Caglayan, Ozan and Madhyastha, Pranava and Specia, Lucia and Barrault, Loïc},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1422},
  url = {https://doi.org/10.18653/v1/n19-1422}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Computing Research Repository, arXiv:1409.0473. Version 7. find our card
  2. Loı̈c Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 308–327, Belgium, Brussels. Ass… find our card
  3. Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer. 2017a. LIUM-CVC submissions for WMT17 multimodal translation task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Sha… find
  4. Ozan Caglayan, Adrien Bardet, Fethi Bougares, Loı̈c Barrault, Kai Wang, Marc Masana, Luis Herranz, and Joost van de Weijer. 2018. LIUM-CVC submissions for WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 603–608, Be… find
  5. Ozan Caglayan, Loı̈c Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine translation. Computing Research Repository, arXiv:1609.03976. — [arXiv:1609.03976](https://arxiv.org/abs/1609.03976) find
  6. Ozan Caglayan, Mercedes Garcı́a-Martı́nez, Adrien Bardet, Walid Aransa, Fethi Bougares, and Loı̈c Barrault. 2017b. NMTPY: A flexible toolkit for advanced neural machine translation systems. Prague Bull. Math. Linguistics, 109:15–28. find
  7. Iacer Calixto, Desmond Elliott, and Stella Frank. 2016. DCU-UvA multimodal MT system report. In Proceedings of the First Conference on Machine Translation, pages 634–638, Berlin, Germany. Association for Computational Linguistics. find
  8. Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 992–1003, Copenhagen, Denmark. Association for Computational Linguistics. find
  9. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  10. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Pap… find
  11. Francis B. Colavita. 1974. Human sensory dominance. Perception & Psychophysics, 16 (2):409–412. find
  12. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248– 255. find
  13. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380. Association for Computational Linguistics. find our card
  14. Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2974–2978, Brussels, Belgium. Association for Computational Linguistics. find
  15. Desmond Elliott, Stella Frank, Loı̈c Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 215– 233… find our card
  16. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74, Berlin, Germany. Association for Computational Linguistics. find our card
  17. Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018. The MeMAD submission to the WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Tran… find
  18. K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. find
  19. Jindřich Helcl, Jindřich Libovický, and Dusan Varis. 2018. CUNI system for the WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 622–629, Belgium, Brussels. Association for Computational Linguistics. find
  20. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based multimodal neural machine translation. In Proceedings of the First Conference on Machine Translation, pages 639–645, Berlin, Germany. Association for Computational Linguistics. find our card
  21. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Computing Research Repository, arXiv:1412.6980. find
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  23. Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia. 2018. Sheffield submissions for WMT18 multimodal translation shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 630–637, Belgium, Brussels. Association for Compu… find
  24. Jindřich Libovický and Jindřich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196–202, Vancouver, Canada. Association for Computational Li… find
  25. Mingbo Ma, Dapeng Li, Kai Zhao, and Liang Huang. 2017. OSU multimodal machine translation system report. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 465–469, Copenhagen, Denmark. Association for Computational Linguistics. find
  26. Pranava Swaroop Madhyastha, Josiah Wang, and Lucia Specia. 2017. Sheffield MultiMT: Using object posterior predictions for multimodal machine translation. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 470–476, Copenhagen, Denmark. Association for… find
  27. Ofir Press and Lior Wolf. 2016. Using the output embedding to improve language models. Computing Research Repository, arXiv:1608.05859. find our card
  28. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of… find our card
  29. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation, pages 543–553, Berlin, Germany. Association for Computational Linguistics. find our card
  30. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929– 1958. find
  31. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/caglayan-2019-probing-need-visual-context-multimodal/