Elamur Research⌕
☰
home / papers / 2022 / Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

Paper · 2022 · Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen et al.

Summary

Multilingual image captioning has lacked reliable evaluation data. Existing benchmarks are mostly English, and translated test sets weaken automatic metrics such as CIDEr, so trustworthy comparisons have required costly human studies that are hard to replicate. The authors built XM3600, a benchmark of 3600 images from Open Images with human-written captions in 36 languages. To make the images geographically diverse, a greedy algorithm used GPS metadata to pick 100 photos per language from areas where that language is spoken. Bilingual annotators first rated English captions produced by a model. This step showed them the target descriptive style. They then wrote captions in their own language without seeing the English text, which kept the style consistent across languages and avoided translation artefacts. Using XM3600 references, CIDEr rankings of captioning model variants agreed well with human side-by-side judgements. This makes XM3600 a reusable, openly licensed (CC-BY 4.0) substitute for much of the human evaluation in multilingual captioning research.

Key points

  • The 36 languages are made up of three groups:

    • 30 languages chosen by web-content share (Ukrainian is one of them),
    • 5 low-resource languages: Bengali, Cusco Quechua, Maori, Swahili and Telugu,
    • English.

    Together they yield 261,375 captions, mostly two independent captions per image. Bengali has one caption per image, and Maori has one or two.

  • Image selection falls back to country, continent and then global level when local photos run short. Persian needed 14 continent-level images. All Hindi images came from the global pool because in-region photos went to Bengali and Telugu.

  • Annotation sessions use batches of 15 images. This batch size was chosen so annotators would not remember the exact wording of the English captions.

  • The validation compared human judgements with CIDEr on XM3600 for English, Spanish, Hindi and Chinese, and 28 more languages were checked. Human data and model outputs were not released.

  • Acknowledged limitations:

    • European languages are over-represented.
    • There are only 100 images per language.
    • Some images may have been taken by tourists.
    • Metrics may miss small differences between models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 worksVision & language · 269 works

Cite

Cite as Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen and Radu Soricut. 2022. Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.45

@inproceedings{thapliyal2022crossmodal,
  title = {{Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset}},
  author = {Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen and Radu Soricut},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.45},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.45}
}

References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. Cross-modal coherence modeling for caption generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6525– 6535, Online. Association for Computational Linguistics. find
  2. Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304–323.
  3. Douglas Bonett and Thomas Wright. 2000. Sample size requirements for estimating pearson, kendall and spearman correlations. Psychometrika, 65:23– 28. find
  4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00325.
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.
  6. Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 215– 233,…
  7. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74. Association for Computational Linguistics.
  8. Markus Freitag, David Grangier, and Isaac Caswell. 2020. Bleu might be guilty but references are not innocent. arXiv, abs/2004.06063.
  9. Andrej Karpathy and Li Fei-Fei. 2014. Deep visual-semantic alignments for generating image descriptions.
  10. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 123(1):32–73.
  11. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship d… find
  12. Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. find
  13. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1179–1195. IEEE Computer Society.
  14. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.
  15. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. find
  16. Ashish V. Thapliyal and Radu Soricut. 2020. Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 160–170, Online. Association for Computational Linguistics. find
  17. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113–123, Brussels, Belgium. Association for Computational Lin…
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
  19. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR.
  20. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  21. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. 2:67–78. find
  22. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2021. Scaling vision transformers. find
  23. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73– 81, Florence, Italy. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thapliyal-2022-crossmodal-3600-massively-multilingual-multimodal/