Elamur Research
☰
home / papers / 2021 / LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2111.02114

Summary

Language-vision models such as CLIP and DALL-E learn from hundreds of millions of image-caption pairs. Before this work, no openly available dataset came close to that size, so only groups with private data could train such models from scratch. The LAION community fills this gap with LAION-400M. The authors pulled HTML image tags that have alt-text out of Common Crawl WAT files and downloaded the images. They then removed pairs with very short captions or tiny files, and dropped duplicates using a bloom filter on URL and alt-text. They also discarded pairs whose CLIP image and text embeddings had cosine similarity below 0.3, and used CLIP to tag NSFW material and remove illegal content. The release contains metadata in parquet files, CLIP embeddings, kNN indices for similarity search, the img2dataset downloader and a web search demo. To show the data is usable, they trained a DALLE-pytorch model on roughly 7.2M samples for one epoch; it converged quickly and produced plausible images. With this release, research on large multimodal models is no longer limited to a few organisations.

Key points

  • LAION-400M has about 413M unique image-text pairs collected from Common Crawl. Each pair has metadata: URL, Creative Commons license type, CLIP-based NSFW tag (under 1% flagged), similarity score and image size.
  • Filtering steps: alt-text under 5 characters or image under 5 KB is dropped, duplicates are removed with a bloom filter, and pairs with CLIP cosine similarity below 0.3 are discarded (the cutoff was chosen by human inspection).
  • About 26M images have height or width of at least 1024 px, and 211M have both sides of at least 256 px, so subsets at different resolutions are possible.
  • With img2dataset, one machine (1 Gbps, 16-core i7, 32 GB RAM) can fetch about 100M images in 20 hours and save them in webdataset format.
  • DALL-E check: DALLE-pytorch with an ImageNet-pretrained VQGAN, where CLIP ViT-B/16 picks the top 8 of 128 samples. It was also compared with runs on Conceptual Captions 3M/12M.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 152 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.02114 · arXiv 2111.02114

@misc{schuhmann2021laion,
  title = {{LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs}},
  author = {Schuhmann, Christoph and Vencu, Richard and Beaumont, Romain and Kaczmarczyk, Robert and Mullis, Clayton and Katta, Aarush and Coombes, Theo and Jitsev, Jenia and Komatsuzaki, Aran},
  year = {2021},
  doi = {10.48550/arxiv.2111.02114},
  eprint = {2111.02114},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.02114}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv e-prints, page arXiv:2103.00020, Febr… find our card
  2. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. arXiv e-prints, page arXiv:2102.12092, February 2021. find our card
  3. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv e-prints, page arXiv:2102.05918, February 2021. find our card
  4. Aran Komatsuzaki. One Epoch Is All You Need. arXiv e-prints, page arXiv:1906.06669, Jun 2019. — [arXiv:1906.06669](https://arxiv.org/abs/1906.06669) find
  5. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv e-prints, page arXiv:2001.08361, Jan 2020. find
  6. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Auto… find
  7. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 491–507, Cham… find
  8. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021. — [arXiv:2106.04560](https://arxiv.org/abs/2106.04560) find
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  10. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv e-prints, page arXiv:2101.00027, December 2020. find
  11. Phil Wang. Dall-e in pytorch: A text to image transformer, 2021. find
  12. Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020. find