🧑‍🔬 Alex Ray

Alex Ray works on large language models, with papers here from 2021–2022 in Language models & pretraining and Evaluation & benchmarks. The work asks two connected questions: how to judge what a large model can actually do, and how to make its outputs match what users want. The methods are fine-tuning pretrained GPT models on targeted data, whether public source code or human-written demonstrations and preference judgments, and then testing them on purpose-built tasks.

The most cited paper here is Training language models to follow instructions with human feedback. It starts from the observation that scaling a model up does not by itself stop it from inventing facts, producing toxic text or ignoring the request. The authors first fine-tune GPT-3 on example answers written by labelers. They then train a reward model on the labelers' rankings of candidate responses and optimise the language model against that reward with PPO, an approach known as reinforcement learning from human feedback. The other main paper, Evaluating Large Language Models Trained on Code, introduces Codex, GPT models fine-tuned on public GitHub code. Instead of scoring generated code by its textual overlap with a reference, it measures functional correctness. It does this with HumanEval, a hand-written set of Python problems checked by unit tests, and reports results as pass@k.

The newer of these papers moves from evaluating what models can do toward alignment, meaning steering a general model's behaviour with direct human feedback rather than more scale alone. The instruction-following work was carried out by a team at OpenAI. Co-authors on Ray's papers here include Long Ouyang, John Schulman, Mark Chen, Jerry Tworek, Jeff Wu, Jacob Hilton, Jared Kaplan and Greg Brockman.

Alex Ray has 2 papers in our library, in 2021–2022, most of them on Language models & pretraining, Evaluation & benchmarks; often with Chong Zhang, Diogo Almeida, Greg Brockman.