Elamur Research
☰
home / papers / 2016 / Multi30K: Multilingual English-German Image Descriptions

Multi30K: Multilingual English-German Image Descriptions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1605.00459

Summary

Research on automatic image captioning has relied almost entirely on English data, leaving other languages without resources for tasks such as caption generation or natural-language image search. Elliott, Frank, Sima'an and Specia address this gap by adding German text to the Flickr30K images, creating Multi30K. The resource has two parts with different cross-lingual relationships. In the first, professional translators converted one English caption per image into German without seeing the picture, which yields sentence-aligned pairs. In the second, German-speaking crowdworkers wrote five new captions per image using a German version of the original collection interface. These captions describe the same image as the English ones but are not aligned with them sentence by sentence. Quality was checked by manual inspection, length and repetition filters, and a character-level 6-gram LM that flagged high-perplexity output. The paper reports corpus statistics, not model experiments. Translated pairs are similar in length, while independent German captions are shorter than English ones and have a larger vocabulary. Multi30K is larger than earlier multilingual image–text sets and supports multimodal MT and multilingual captioning.

Key points

  • Multi30K extends Flickr30K (31,014 images) with 31,014 professional German translations and 155,070 independently crowdsourced German descriptions.
  • Translators worked without the images, at a total cost of about €23,000 (€0.06 per word). The English sources were chosen in equal shares of short, median and long captions.
  • 185 Crowdflower workers wrote the descriptions over 31 days and were paid $0.05 per caption. Total spending, including platform fees, was $9,591.24.
  • The German texts are lexically richer than the English ones:
    • Average length is 11.1 words for German translations versus 9.6 for German descriptions.
    • German vocabularies are more than twice the size of the English ones, with two to three times as many singletons, likely due to morphology and compounding.
  • Earlier multilingual image datasets were smaller and translation-only: IAPR-TC12 has 20,000 English-German images, and Pascal Sentences has 1,000 Japanese-English images.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

257 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 249 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1605.00459 · arXiv 1605.00459

@misc{elliott2016multi30k,
  title = {{Multi30K: Multilingual English-German Image Descriptions}},
  author = {Elliott, Desmond and Frank, Stella and Sima'an, Khalil and Specia, Lucia},
  year = {2016},
  doi = {10.48550/arxiv.1605.00459},
  eprint = {1605.00459},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.00459}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. find our card
  2. Raffaella Bernardi, Ruken Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 5… find
  3. Iacer Calixto, Teo E. de Campos, and Lucia Specia. 2012. Images as context in statistical machine translation. In Second Annual Meeting of the EPSRC Network on Vision & Language, Sheffield. find
  4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. find our card
  5. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263–270. Association for Computational Linguistics. find our card
  6. Desmond Elliott and Frank Keller. 2013. Image Description using Visual Dependency Representations. In EMNLP, pages 1292– 1302. find
  7. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. CoRR, abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) find
  8. Ruka Funaki and Hideki Nakayama. 2015. Image-mediated learning for zero-shot cross-lingual document retrieval. In Empirical Methods in Natural Language Processing 2015, pages 585–590. find
  9. Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Desealers. 2006. The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In LREC. find
  10. Julian Hitschler and Stefan Riezler. 2016. Multimodal pivots for image caption translation. CoRR, abs/1601.03916. find
  11. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. Journal of Artificial Intelligence Research, 47:853–899. find
  12. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. As… find
  13. Cyrus Rashtchian, Peter Young, Micha Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In NAACLHLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  14. Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. find
  15. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. find our card
  16. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044. find our card
  17. Peter Young, Alice Lai, Micha Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics. find