🧑‍🔬 Siddhartha Brahma

Siddhartha Brahma's papers here concern large language models and how they learn to follow instructions. They ask how models can be trained to handle tasks they have not seen, and how instruction-following can be measured reliably. The methods include finetuning pretrained models on broad mixtures of instruction-phrased tasks and designing benchmarks whose outcomes can be checked automatically. This places the work within language models and pretraining and evaluation and benchmarks.

The most cited of these papers is Scaling Instruction-Finetuned Language Models from 2022. It extends the Flan procedure along three axes: the number of training tasks, the size of the model, and the inclusion of step-by-step reasoning examples. The approach is applied to T5, PaLM and U-PaLM at a wide range of scales to see how each axis affects generalization. The second work, Instruction-Following Evaluation for Large Language Models from 2023, introduces IFEval. This benchmark is built around "verifiable instructions," constraints such as length limits, letter case or output format that a simple program can test. It avoids the cost of human raters and the biases of model-based judges.

The newest work thus moves from training models to follow instructions toward checking whether they actually do. Bylines from 2020 to 2023 list Google. Earlier papers list IBM Research - Almaden and, from 2011 to 2016, École Polytechnique Fédérale de Lausanne. Co-authors include Le Hou, Hyung Won Chung and Denny Zhou, along with others on the Flan scaling study.

Siddhartha Brahma has 2 papers in our library, in 2022–2023, most of them on Language models & pretraining, Evaluation & benchmarks; often with Le Hou, Albert Webson, Barret Zoph.