Elamur Research⌕
☰
home / papers / 2021 / WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

Paper · 2021 · Krishna Srinivasan, Karthik Raman, Jiecao Chen et al.

Summary

Public image-text datasets used to train multimodal models are much smaller than text-only or image-only corpora, and almost all of them are English-only. This limits multilingual vision-language research. The authors address this with WIT, a dataset built from Wikipedia content pages and their Wikimedia-hosted images. For each image they pull several associated texts: reference, attribution and alt descriptions, plus page and section context. They then apply filters to drop generic or low-information pairings.

Because Wikipedia is edited and reviewed by people, the data is cleaner than alt-text scraped from web crawls such as Conceptual Captions. Human raters confirmed this, judging nearly all sampled pairs to be good matches.

The authors trained a simple dual encoder on WIT. It retrieves images far better than a CC-trained baseline on Wikipedia-based test sets, though it transfers less well to Flickr30K and MS-COCO. They attribute the gap to WIT's entity-heavy, specific vocabulary, which contrasts with the short, generic captions in those benchmarks.

WIT works both as a large pretraining source and as a hard, realistic multilingual benchmark.

Key points

  • Size and coverage: WIT holds 37.6M image-text examples with 11.5M unique images across 108 languages. Every language has at least 12K examples, and 53 languages exceed 100K. At the time of writing it was about 3x larger than any other public multimodal dataset.
  • Source data and splits: The starting point was about 124M Wikipedia pages in 279 languages, processed with a Flume pipeline into roughly 150M candidate tuples before filtering. The final splits are 37.13M train, 261.8K validation and 210.7K test.
  • Quality check: Human annotation covered about 4,400 examples with 13K judgments in 7 languages. Raters judged 98.5% of the pairs favorably.
  • Why transfer is weaker: About 72% of WIT unigrams appear three times or fewer, compared with 43% for CC. Of roughly 4.5M image entities, at least 80% occur three times or fewer. The authors also note that WIT lacks the hypernymization step used in CC.
  • Availability and venue: The dataset is released at github.com/google-research-datasets/wit, and the paper was presented at the SIGIR 2021 Resource Track.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksVision & language · 811 works

Cite

Cite as Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky and Marc Najork. 2021. WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning. arXiv:2103.01913.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3404835.3463257 · arXiv 2103.01913

@misc{srinivasan2021wit,
  title = {{WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning}},
  author = {Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky and Marc Najork},
  year = {2021},
  doi = {10.1145/3404835.3463257},
  eprint = {2103.01913},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3404835.3463257}
}

References

43 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. 2019. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv:1908.05054 (2019). find
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433. find
  3. Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In THIRD CONFERENCE ON MACHINE TRANSLATION (WMT18), Vol. 2. 308–327.
  4. Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R Henry, Robert Bradshaw, and Nathan Weizenbaum. 2010. FlumeJava: easy, efficient data-parallel pipelines. ACM Sigplan Notices 45, 6 (2010), 363–375. find
  5. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. arXiv preprint arXiv:2102.08981 (2021). — [arXiv:2102.08981](https://arxiv.org/abs/2102.08981) find
  6. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 (2019).
  7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  9. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020…
  10. Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. arXiv preprint arXiv:1710.07177 (2017).
  11. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. arXiv preprint arXiv:1605.00459 (2016).
  12. John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018. Learning translations via images with a massively multilingual image dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  13. Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal pivots for image caption translation. arXiv preprint arXiv:1601.03916 (2016). find
  14. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research 47 (2013), 853–899. find
  15. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv preprint arXiv:2102.05918 (2021).
  16. Da-Cheng Juan, Chun-Ta Lu, Zhen Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. 2019. Graph-rise: Graph-regularized image semantic embedding. arXiv preprint arXiv:1902.10814 (2019). — [arXiv:1902.10814](https://arxiv.org/abs/1902.10814) find
  17. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. 2019. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. arXiv e-prints, page. arXiv preprint arXiv:1908.06066 (2019). find
  18. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. 2020. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training.. In AAAI. 11336–11344. find
  19. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019).
  20. Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. 2019. COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval. IEEE Transactions on Multimedia 21, 9 (2019), 2347–2360. find
  21. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
  22. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems. 13–23.
  23. Takashi Miyazaki and Nobuyuki Shimizu. 2016. Cross-Lingual Image Caption Generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1780–1790. https://doi.org/10.18653/v1/P16-1168 ↗
  24. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems 24 (2011), 1143–1151. find
  25. Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. 2020. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 (2020). — [arXiv:2001.07966](https://arxiv.org/abs/2001.07966) find
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. Image 2 (2021), T2.
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019).
  28. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2556–2565.
  29. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  30. Karan Singhal, Karthik Raman, and Balder ten Cate. 2019. Learning multilingual word embeddings using image-text data. arXiv preprint arXiv:1905.12260 (2019). find
  31. Lucia Specia, Stella Frank, Khalil Sima’An, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers. 543–553.
  32. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019).
  33. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. arXiv:cs.CV/1707.02968
  34. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2818–2826. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017), 5998–6008.
  36. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3156–3164.
  37. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. arXiv preprint arXiv:1904.09077 (2019).
  38. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv:cs.CL/2010.11934
  39. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67–78. find
  40. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6720–6731. find
  41. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. 2020. Unified Vision-Language Pre-Training for Image Captioning and VQA.. In AAAI. 13041–13049. find
  42. Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu. 2018. A visual attention grounding neural model for multimodal machine translation. arXiv preprint arXiv:1808.08266 (2018). find
  43. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision. 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srinivasan-2021-wit-wikipedia-based-image-text/