🧑‍🔬 Prafulla Dhariwal

Prafulla Dhariwal works on large generative models in machine learning. Their papers here span language models and pretraining and vision and language. A recurring question is how autoregressive Transformers and related generative systems behave as model size and compute grow, and how such models can produce text or images from a prompt. The work is empirical: it trains families of models and evaluates them across many benchmarks and data domains.

The most cited of these papers is Language Models are Few-Shot Learners (2020). It introduced GPT-3 and showed that a very large autoregressive language model can take on new tasks from instructions and a handful of worked examples placed in its prompt, with no further training. The authors named this ability in-context learning. A related study, Scaling Laws for Autoregressive Generative Modeling (2020), asked whether the smooth relationship between loss, model size and compute seen for text also holds elsewhere. It found that the relationship holds for images, video, image–caption pairs and synthetic mathematics problems alike.

The newest work turns toward image synthesis. Hierarchical Text-Conditional Image Generation with CLIP Latents (2022) describes unCLIP. In this two-stage approach, a prior first predicts a CLIP image embedding from a caption, and a diffusion decoder then renders that embedding as a picture. Further diffusion stages raise the image to high resolution. The study found that a diffusion-based prior was both cheaper and better than an autoregressive one. Dhariwal's papers from 2021 to 2022 list OpenAI as the affiliation. Collaborators include Tom B. Brown, Jared Kaplan, Benjamin Mann, Mark Chen and Aditya Ramesh.

Prafulla Dhariwal has 3 papers in our library, in 2020–2022, most of them on Language models & pretraining, Vision & language; often with Benjamin Mann, Jared Kaplan, Mark Chen.