🧑‍🔬 Samir Yitzhak Gadre

Samir Yitzhak Gadre works on how large pretrained neural networks are built, adapted and evaluated, in both language models & pretraining and vision & language. A thread through their papers here is making large-scale training open and measurable. The papers release training code, hold most of a pipeline fixed, and isolate a single ingredient, such as model weights or training data, to see what it contributes.

The most cited of these papers is Model soups (2022). It shows that the many runs from a fine-tuning hyperparameter sweep need not be thrown away. When several networks start from one shared pretrained checkpoint, their parameters can be averaged into a single model. That model gains accuracy and is no more expensive to run than any individual member. OpenFlamingo (2023) offers an open-source framework for training autoregressive vision-language models. These models read interleaved images and text and can learn in context from a few examples. The framework was built as an openly available counterpart to proprietary systems such as DeepMind's Flamingo, which keep their weights, data and code closed.

The newest work turns toward the training data itself. DataComp-LM (2024) is a benchmark that keeps model and training settings constant, so the dataset is the only thing that varies. It offers a pool of 240T tokens gathered from Common Crawl, along with code for training, model sizes ranging between 412M and 7B parameters, and a suite of 53 evaluations. In its baseline experiments, filtering text with a model that judges quality proved the most important choice.

From 2021 to 2024, Gadre's papers list Columbia University in New York. The work is collaborative, and Simon Kornblith appears on two of the papers here. Other co-authors include Ludwig Schmidt, Ali Farhadi and Gabriel Ilharco.

Samir Yitzhak Gadre has 3 papers in our library, in 2022–2024, most of them on Language models & pretraining, Vision & language; often with Simon Kornblith, Alex Fang, Ali Farhadi.