🧑‍🔬 Pranav Shyam

Pranav Shyam works on large neural language models and how they can be pretrained to serve many tasks at once. Shyam's papers here, from 2020 to 2022, fall under Language models & pretraining. They share one question: whether a single model trained on broad, unlabelled text from the web, without task-specific datasets or architectures, can handle a wide range of downstream uses. The methods are large Transformer models trained on web-scale corpora and tested across many benchmarks.

The most cited of these is Language Models are Few-Shot Learners, the GPT-3 paper. It asked whether making an autoregressive model much larger lets it pick up new language tasks the way people do, from a short instruction or a handful of examples. The model was evaluated with no weight updates at all, with each task and its demonstrations written directly into the prompt. The authors named this behaviour in-context learning. The second paper, Text and Code Embeddings by Contrastive Pre-Training, applies a similar general-purpose approach to vector representations. It starts from generative pretrained weights, GPT for text and Codex for code. It then trains an encoder contrastively, using unlabelled pairs that arise on their own in existing material: neighbouring passages of web text, and docstrings paired with their code. One recipe then serves semantic search, classification and similarity scoring alike.

The newer of the two works moves from generating text to producing embeddings that span both natural language and source code. It brings sentence-embedding research and neural retrieval under one training scheme. Shyam's co-authors include Arvind Neelakantan, who appears on both papers, as well as Jared Kaplan, Alec Radford and Amanda Askell.

Pranav Shyam has 2 papers in our library, in 2020–2022, most of them on Language models & pretraining; often with Arvind Neelakantan, Alec Radford, Amanda Askell.