Elamur Research
☰
home / papers / 2016 / A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2346.pdf

Summary

Most joint language-and-vision research, such as image captioning, has worked with English only. This paper reports on the first shared task designed to bring multilinguality into the field, organised alongside WMT16. Both variants use English and German and the Multi30K dataset, which extends Flickr30K.

  • Task 1 (Multimodal Machine Translation): systems translate an English image caption into German and may draw on the picture.
  • Task 2 (Crosslingual Image Description): systems produce a German description of an image and may use English captions as extra input.

The organisers released VGG-19 image features and provided baselines: a Moses phrase-based SMT system and the GroundedTranslation model. Ten teams entered 23 systems in total.

The main outcome is that images added little. In Task 1, visual signals helped only marginally and indirectly, through reranking. In Task 2, a text-only neural system achieved the top score. This shows how difficult it is to make visual input useful and motivates more robust multimodal models and harder datasets.

Key points

  • Participation: 10 teams submitted 16 systems for Task 1 and seven for Task 2.
  • Data: Multi30K covers 31,014 Flickr30K images.
    • Task 1 trains on about 29,000 professionally translated sentence pairs and is scored against one reference.
    • Task 2 trains on about 145,000 crowdsourced descriptions (via Crowdflower) and is scored against five independent German references.
  • Released visual features came from the VGG-19 FC 7 (relu7) and CONV 5,4 layers.
  • Task 2 results:
    • The winner was LIUM 2 TextNMT C, which used no image features (Meteor 35.1, BLEU 23.8).
    • LIUM's ResNet-50-based multimodal system came second (Meteor 32.3).
    • The GroundedTranslation baseline scored Meteor 31.2.
  • English→German translation quality was surprisingly high given the small training set. The authors attribute this to simple, repetitive captions, unlike WMT News data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2346

@inproceedings{specia2016shared,
  title = {{A Shared Task on Multimodal Machine Translation and Crosslingual Image Description}},
  author = {Specia, Lucia and Frank, Stella and Sima'an, Khalil and Elliott, Desmond},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2346},
  url = {https://doi.org/10.18653/v1/w16-2346}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Desmond Elliott and Frank Keller. 2014. Comparing Automatic Evaluation Measures for Image Description. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 452–457, Baltimore, Maryland. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  3. Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. CoRR, abs/1601.03896. — [arXiv:1601.03896](https://arxiv.org/abs/1601.03896) find
  4. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. CoRR, abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) find
  5. Desmond Elliott, Stella Frank, Khalil. Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German Image Descriptions. In Proceedings of the 5th Workshop on Vision and Language, Berlin, Germany. find our card
  6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc… find our card
  7. Yansong Feng and Mirella Lapata. 2010. Topic models for image annotation and text illustration. In Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 831–839, Los Angeles, California. find
  8. Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, and Joost van de Weijer. 2016. Does multimodality help human and machine for translation and image captioning? In Proceedings of the First Conference on Machine Translation, Berlin, Ger… find
  9. Ruka Funaki and Hideki Nakayama. 2015. Image-mediated learning for zero-shot cross-lingual document retrieval. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 585–590, Lisbon, Portugal. find
  10. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question answering. Advances in Neural Information Processing Systems, pages 2287–2295. find
  11. Iacer Calixto, Desmond Elliott, and Stella Frank. 2016. Dcu-uva multimodal mt system report. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. find
  12. Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Desealers. 2006. The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In Proceedings of the Language Resources and Evaluation Conference. find
  13. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Pap… find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. CoRR, abs/1512.03385. find
  15. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal Pivots for Image Caption Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. find
  16. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland. find our card
  17. Chris Hokamp and Iacer Calixto. 2016. Multimodal neural machine translation using minimum risk training. https://www.github.com/c hrishokamp/multimodal\_nmt. ↗
  18. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations. find our card
  19. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based multimodal neural machine translation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. find
  20. Matthew Snover, Bonnie Dorr, and Richard Schwartz. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, Cambridge, Massachusetts. find our card
  21. Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21. find
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual meeting of Association for Comp… find
  23. Darlene Stewart, Roland Kuhn, Eric Joanis, and George Foster. 2014. Coarse split and lump bilingual language models for richer source information in SMT. In Proceedings of the Eleventh Conference of the Association for Machine Translation in the Americas, pages 28–41, Vancouver, Canada. find
  24. Jindřich Libovický, Jindřich Helcl, Marek Tlustý, Ondřej Bojar, and Pavel Pecina. 2016. Cuni system for wmt16 automatic post-editing and multimodal translation tasks. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. find
  25. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In The IEEE Conference on Computer Vision and Pattern Recognition. find our card
  26. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2015. Stacked attention networks for image question answering. CoRR, abs/1511.02274. find
  27. Kishore Papineni, Salim Roukos, Todd Ard, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. find our card
  28. Peter Young, Alice Lai, Micha Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  29. Arnau Ramisa, Fei Yan, Francesc Moreno-Noguer, and Krystian Mikolajczyk. 2016. Breakingnews: Article annotation by image and text processing. CoRR, abs/1603.07141. — [arXiv:1603.07141](https://arxiv.org/abs/1603.07141) find
  30. Sergio Rodrı́guez Guasch and Marta R. Costa-jussà. 2016. find
  31. Wmt 2016 multimodal translation system description based on bidirectional recurrent neural networks with double-embeddings. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. find
  32. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. 2014. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575. find our card
  33. Kashif Shah, Josiah Wang, and Lucia Specia. 2016. Shef-multimodal: Grounding machine translation on images. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. find
  34. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. find our card