Elamur Research⌕
☰
home / papers / 2018 / Findings of the Third Shared Task on Multimodal Machine Translation

Findings of the Third Shared Task on Multimodal Machine Translation

Summary

This overview describes the third WMT shared task on multimodal machine translation (2018). Systems translate English image captions into German, French or Czech, and they may use the paired image. The organisers added Czech to the Multi30K dataset, making it 5-way aligned. They also introduced Task 1b, a multisource variant in which parallel English, French and German captions are translated into Czech.

New test material came from 2,071 manually filtered Flickr images. Participants received ResNet-50 visual features. Alongside Meteor, BLEU, TER and human direct assessment, a new metric, Lexical Translation Accuracy (LTA), measures how well systems translate ambiguous source words. It relies on human-curated lists of context-appropriate translations.

Seven teams entered 45 systems. Multimodal submissions improved over the previous year, but text-only systems stayed competitive. LTA revealed one advantage that other metrics did not show: multimodal variants tended to handle ambiguous words better than their text-only versions. The task matters because it offers a shared benchmark and new resources for testing whether visual context helps translation.

Key points

  • Czech translations of Multi30K:
    • 15 translators produced them for about EUR 3,500.
    • Automatic checks flagged 5,255 segments (16%), which were then manually fixed.
    • A quality sample found 94% free of spelling errors, 96% preserving meaning and 75% judged fluent.
  • Splits and features: the data has 29,000 training and 1,014 development tuples. Both 'res4 relu' convolutional and average-pooled ResNet-50 features were released.
  • LTA dictionaries: these were built with fast align, SECOS (German) and MorphoDiTa (Czech). The English-Czech dictionary skipped human filtering and is therefore noisier.
  • Human evaluation leaders: MeMAD-OpenNMT-mmod (unconstrained) led English→French, and CUNI NeuralMonkeyImagination led English→Czech. Ensembling helped both text-only and multimodal systems.
  • Task 1b: only two teams took part, and human judgments could not separate the systems into clusters. A wide gap to the human references remains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

86 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6402

@inproceedings{barrault2018findings,
  title = {{Findings of the Third Shared Task on Multimodal Machine Translation}},
  author = {Barrault, Loïc and Bougares, Fethi and Specia, Lucia and Lala, Chiraag and Elliott, Desmond and Frank, Stella},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6402},
  url = {https://doi.org/10.18653/v1/w18-6402}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  2. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, pages 1171–1179. MIT Press. find
  3. Ozan Caglayan, Adrien Bardet, Fethi Bougares, Loı̈c Barrault, Kai Wang, Marc Masana, Luis Herranz, and Joost van de Weijer. 2018. LIUM-CVC submissions for WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium… find
  4. Ozan Caglayan, Loı̈c Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine translation. CoRR, abs/1609.03976. — [arXiv:1609.03976](https://arxiv.org/abs/1609.03976) find
  5. Ozan Caglayan, Mercedes Garcı́a-Martı́nez, Adrien Bardet, Walid. Aransa, Fethi. Bougares, and Loı̈c Barrault. 2017. NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems. CoRR, 1706.00457. find
  6. Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176–181. find
  7. Jean-Benoit Delbrouck and Stéphane Dupont. 2018. Umons submission for wmt18 multimodal translation task. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  8. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In EACL 2014 Workshop on Statistical Machine Translation. find our card
  9. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2013), pages 644–649. find
  10. Desmond Elliott, Stella Frank, Loı̈c Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 215–233,… find our card
  11. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. CoRR, abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) find
  12. Desmond Elliott, Stella Frank, Khalil Simaan, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In 5th Workshop on Vision and Language, pages 70–74. find our card
  13. Desmond Elliott and Ákos Kádár. 2017. Imagination improves Multimodal Translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 130–141, Taipei, Taiwan. find
  14. Christian Federmann. 2012. Appraise: An open-source toolkit for manual evaluation of machine translation output. The Prague Bulletin of Mathematical Linguistics, 98:25–35. find
  15. Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. 2018. Detectron. https://github.com/ facebookresearch/detectron. ↗
  16. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2015. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  17. Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018. The MeMAD submission to the WMT18 multimodal translation task. In Proceedings of the Third Conference on Machine Tran… find
  18. Jeremy Gwinnup, Joshua Sandvick, Michael Hutt, Grant Erdmann, John Duselis, and James Davis. 2018. The afrl-ohio state wmt18 multimodal system: Combining visual with traditional. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguisti… find
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778. find
  20. Jindrřich Helcl, Jindřich Libovický, and Dušan Variš. 2018. CUNI system for the WMT18 multimodal translation tasks. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  21. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal Pivots for Image Caption Translation. In 54th Annual Meeting of the Association for Computational Linguistics, pages 2399– 2409. find
  22. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, Syste… find our card
  23. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In 45th Annual meeting of Association for Computational Linguisti… find
  24. Chiraag Lala, Pranava Madhyastha, Carolina Scarton, and Lucia Specia. 2018. Sheffield submissions for wmt18 multimodal translation shared task. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  25. Chiraag Lala and Lucia Specia. 2018. Multimodal Lexical Translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. find
  26. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. CoRR, abs/1405.0312. find our card
  27. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). find our card
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. find our card
  29. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1156–1167. find
  30. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1179–1195, Honolulu, Hawaii. find our card
  31. Martin Riedl and Chris Biemann. 2016. Unsupervised compound splitting with distributional semantics rivals supervised methods. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 617–622. find
  32. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas. find our card
  33. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In First Conference on Machine Translation, pages 543–553. find our card
  34. Jana Straková, Milan Straka, and Jan Hajič. 2014. Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity Recognition. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 13–18, Baltimore, Maryland. Association… find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  36. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  37. Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. arXiv preprint arXiv:1603.07012. find
  38. Renjie Zheng, Yilin Yang, Mingbo Ma, and Liang Huang. 2018. Ensemble sequence level training for multimodal mt: Osu-baidu wmt18 multimodal machine translation system report. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/barrault-2018-findings-third-shared-task-multimodal/