🧑‍🔬 Ross Wightman

Ross Wightman's papers here belong to Vision & language research, with a focus on training image-text models at very large scale using resources anyone can obtain. A recurring question is how to move this kind of work out of a handful of industry labs, where training sets and models have largely stayed private. The methods combine web-scale collection of paired images and captions, automatic filtering of those pairs, and contrastive language-image pre-training run on the open-source OpenCLIP codebase.

The most cited of these works is LAION-5B, which released a public collection of billions of image-text pairs. The images and their alt-text were drawn from Common Crawl and downloaded in a distributed fashion. Pairs were kept only when an existing CLIP model judged the image and text similar enough. OpenCLIP models trained on a subset showed that the collection works as training material. A companion study, Reproducible scaling laws for contrastive language-image learning, used that public data to measure how CLIP performance changes as the vision encoder, the dataset and the number of training samples seen grow. Because both the data and the code are open, the study can be repeated in full, which earlier scaling work on closed models did not allow.

Both papers date from 2022, and together they point in one direction: open datasets and open models as a shared foundation for multimodal research. The work is collaborative. Recurring co-authors include Christoph Schuhmann, Romain Beaumont, Mehdi Cherti, Mitchell Wortsman and Cade Gordon, with Jenia Jitsev and Ludwig Schmidt among other contributors.

Ross Wightman has 2 papers in our library, in 2022, most of them on Vision & language; often with Cade Gordon, Christoph Schuhmann, Mehdi Cherti.