🧑‍🔬 Shixiang Shane Gu

Shixiang Shane Gu works on large language models, with a focus on how such models can be made to reason through multi-step problems and to handle tasks they were not trained on directly. The papers here, both from 2022, sit within language models and pretraining and reasoning. They share a practical question: how much can be gained from the way a model is prompted or finetuned, rather than from new architectures? The usual method is empirical, testing one approach across many benchmarks and model sizes.

The most cited of these papers, Large language models are zero-shot reasoners, shows that a model can produce step-by-step reasoning without any hand-crafted worked examples. A single generic trigger phrase placed after the question prompts the model to reason aloud, and a second prompt then extracts the final answer. The same template, left unchanged, was evaluated on arithmetic word problems, symbolic puzzles, commonsense question answering and logic tasks. The second paper, Scaling Instruction-Finetuned Language Models, studies finetuning on a very large collection of tasks phrased as instructions. It asks how results change as the number of tasks grows, as models get larger, and when training includes examples with written-out reasoning. It applies this procedure to T5, PaLM and U-PaLM models across a wide range of sizes.

Together the two papers trace a single line of work on chain-of-thought reasoning, approached once through prompting alone and once through instruction finetuning. Co-authors on the finetuning study include Hyung Won Chung, Yi Tay, Shayne Longpre and Mostafa Dehghani. The zero-shot reasoning paper was written with a different group of collaborators.

Shixiang Shane Gu has 2 papers in our library, in 2022, most of them on Language models & pretraining, Reasoning; often with Albert Webson, Barret Zoph, Hyung Won Chung.