🧑‍🔬 Xu Jiang

Xu Jiang's papers here deal with large language models and how to make their behaviour match what people actually ask for. The common thread is learning from human judgement: people write example answers or compare candidate outputs, a reward model is fitted to those preferences, and reinforcement learning then tunes the language model against it. Both works build on GPT-3 and appeared in 2021 and 2022.

The most cited of them, Training language models to follow instructions with human feedback, starts from the observation that scaling a model up does not by itself stop it from inventing facts, producing toxic text or ignoring the request. Hired labelers first wrote demonstration answers to prompts, which were used for supervised fine-tuning. They then ranked several model outputs, a reward model was trained on those rankings, and the policy was optimised with PPO. The earlier WebGPT applies the same feedback-driven training to long-form question answering. GPT-3 is given a simple text-only browser in which it can search, click links, scroll and quote passages, and the quoted passages become citations for its final answer. Training combined behaviour cloning on human browsing demonstrations, a reward model learned from pairwise comparisons of answers, and further reinforcement learning together with rejection sampling.

Taken together, the two papers move from grounding answers in retrieved web sources to the broader goal of aligning a general-purpose model with user intent. Both were written by teams at OpenAI. Co-authors on both include Long Ouyang, Jacob Hilton and Jeff Wu. Other collaborators include Reiichiro Nakano and John Schulman.

Xu Jiang has 2 papers in our library, in 2021–2022, most of them on Language models & pretraining, Question answering; often with Jacob Hilton, Jeff Wu, Long Ouyang.