Elamur Research⌕
☰
home / papers / 2017 / Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Summary

Most image captioning research targets English. This WMT 2017 shared task tests how well systems can produce image descriptions in other languages. It had two subtasks. In Task 1, systems translate an English caption into German or French with the image as extra context. In Task 2, systems must write a German description from the image alone, since no source caption is available at test time. The organisers added French crowdsourced translations to Multi30K, which makes the dataset aligned across three languages. They also built two new test sets: a fresh Multi30K 2017 split drawn from Flickr groups, and Ambiguous COCO, a small set of captions containing verbs whose sense the image may clarify. Nine teams sent 19 systems, and human judgements were collected for translation for the first time. Multimodal systems improved over 2016, but text-only models stayed close in performance. External data helped, especially for sentences with unknown words. Humans often favoured multimodal outputs that automatic metrics did not reward, so the organisers argue that future evaluation should rely on human judgement.

Key points

  • French translations: 12 crowdworkers produced the French translations of Multi30K, at an estimated cost of about €9,700. Their output differs from phrase-based MT suggestions at a TER of 0.41, so it was not simple post-editing.
  • Multi30K 2017 test: 1,000 images for translation and 1,071 images (5,355 German descriptions) for description generation. Organisers also released ResNet-50 visual features ('res4 relu' and average-pooled).
  • Ambiguous COCO: 461 captions covering 56 ambiguous verbs, derived from the VerSe dataset. Visual features gave mixed results on it, and the text-only NICT NMTrerank system placed in the top three for both target languages.
  • Out-of-vocabulary words: 16% of 2017 test sentences contain an OOV word, compared with 1.7% in the earlier Flickr test. On these sentences, the unconstrained UvA-TiCC IMAGINATION system led all metrics.
  • Out-of-domain baseline: A strong WMT'16 news-domain system (Sennrich et al.) landed mid-table (Meteor 49.9, BLEU 29.0). This shows that in-domain tuning still matters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

108 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4718

@inproceedings{elliott2017findings,
  title = {{Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description}},
  author = {Elliott, Desmond and Frank, Stella and Barrault, Loïc and Bougares, Fethi and Specia, Lucia},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4718},
  url = {https://doi.org/10.18653/v1/w17-4718}
}

References

52 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  2. Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. J. Artif. Intell. Res., 55:409–442. find
  3. Ozan Caglayan, Loı̈c Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine translation. CoRR, abs/1609.03976. — [arXiv:1609.03976](https://arxiv.org/abs/1609.03976) find
  4. Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer. 2017a. LIUM-CVC Submissions for WMT17 Multimodal Translation Task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Sha… find
  5. Ozan Caglayan, Mercedes Garcı́a-Martı́nez, Adrien Bardet, Walid. Aransa, Fethi. Bougares, and Loı̈c Barrault. 2017b. NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems. CoRR, 1706.00457. find
  6. Iacer Calixto, Koel Dutta Chowdhury, and Qun Liu. 2017a. DCU System Report on the WMT 2017 Multi-modal Machine Translation Task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 440–444. find
  7. Iacer Calixto, Qun Liu, and Nick Campbell. 2017b. Incorporating Global Visual Features into Attention-Based Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. \#\# Acknowledgements Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Veda… find our card
  8. This work was supported by the CHIST-ERA M2CR project (French National Research Agency Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In 49th Annual Meeting of the Association for… find
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778. find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition. find
  11. Jindřich Helcl and Jindřich Libovický. 2017. CUNI System for the WMT17 Multimodal Translation Task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 450–457. find
  12. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In EACL 2014 Workshop on Statistical Machine Translation. find our card
  13. John Duselis, Michael Hutt, Jeremy Gwinnup, James Davis, and Joshua Sandvick. 2017. The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 445–449. find
  14. Desmond Elliott and Ákos Kádár. 2017. nation improves Multimodal Translation. abs/1705.04350. — [arXiv:1705.04350](https://arxiv.org/abs/1705.04350) find
  15. Imagi-CoRR, Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Short Papers, pages 452–457. find
  16. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. CoRR, abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) find
  17. Desmond Elliott, Stella Frank, Khalil Simaan, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In 5th Workshop on Vision and Language, pages 70–74. find our card
  18. Gertrud Faaß and Kerstin Eckart. 2013. SdeWaC–a corpus of parsable sentences from the web. In Language processing and knowledge in the Web, pages 61–68. Springer. find
  19. Christian Federmann. 2012. Appraise: An open-source toolkit for manual evaluation of machine translation output. The Prague Bulletin of Mathematical Linguistics, 98:25–35, September. find
  20. Spandana Gella, Mirella Lapata, and Frank Keller. 2016. Unsupervised visual sense disambiguation for verbs using multimodal embeddings. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–192, San Diego, California… find
  21. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2015. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  22. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal Pivots for Image Caption Translation. In 54th Annual Meeting of the Association for Computational Linguistics, pages 2399– 2409. find
  23. Alan Jaffe. 2017. Generating Image Descriptions using Multilingual Data. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 458–464. find
  24. Douwe Kiela. 2016. MMFeat: A toolkit for extracting multi-modal features. In Proceedings of ACL-2016 System Demonstrations, pages 55–60. find
  25. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proceedings of EACL 2017, pages 199–209. find
  26. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In 45th Annual meeting of Association for Computational Linguisti… find
  27. D.T. Le, R. Bernardi, and J.R.R. Uijlings. 2014. TUHOI: Trento Universal Human Object Interaction Dataset. In Vision and Language Workshop at the 26th International Conference on Computational Linguistics. find
  28. Xirong Li, Weiyu Lan, Jianfeng Dong, and Hailong Liu. 2016. Adding chinese captions to images. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, pages 271–275. find
  29. Jindřich Libovický and Jindřich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. CoRR, abs/1704.06567. find
  30. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. CoRR, abs/1405.0312. find our card
  31. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. find
  32. Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 65–68. find
  33. Mingbo Ma, Dapeng Li, Kai Zhao, and Liang Huang. 2017. OSU Multimodal Machine Translation System Report. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 465–469. find
  34. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas. find our card
  35. Pranava Swaroop Madhyastha, Josiah Wang, and Lucia Specia. 2017. Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 470–476. find
  36. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In First Conference on Machine Translation, pages 543–553. find our card
  37. Takashi Miyazaki and Nobuyuki Shimizu. 2016. Cross-lingual image caption generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1780–1790. find
  38. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. CoRR, abs/1512.00567. find
  39. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In 41st Annual Meeting on Association for Computational Linguistics, pages 160–167, Sapporo, Japan. find
  40. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Eight International Conference on Language Resources and Evaluation (LREC’12). find
  41. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. find our card
  42. Mesut Erhan Unal, Begum Citamak, Semih Yagcioglu, Aykut Erdem, Erkut Erdem, Nazli Iki-zler Cinbis, and Ruket Cakici. 2016. Tasviret: Görüntülerden otomatik türkçe açıklama oluşturma İçin bir denektaçı veri kümesi (TasvirEt: A benchmark dataset for automatic Turkish description generation… find
  43. Janarthanan Rajendran, Mitesh M Khapra, Sarath Chandar, and Balaraman Ravindran. 2016. Bridge correlational neural networks for multilingual multimodal representation learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  44. Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. 2017. Cross-linguistic differences and similarities in image descriptions. In Proceedings of the 10th International Conference on Natural Language Generation. find
  45. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–25… find our card
  46. Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. 2017. Stair captions: Constructing a large-scale japanese image caption dataset. CoRR, abs/1705.00823. find
  47. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  48. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation. find our card
  50. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2017. NICT-NAIST System for WMT17 Multimodal Translation Task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 477–482, Copenhagen, Denmark. find
  51. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, Berlin, Germany. find
  52. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, et al. 2017. Nematus: a toolkit for neural machine translation. In find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/elliott-2017-findings-second-shared-task-multimodal/