🧑‍🔬 David R. So

David R. So works on large neural network models for natural language. The papers here, from 2020 and 2021, take up two related questions: whether a single generative model trained end to end can hold an open-ended conversation, and how much energy and carbon the training of such models really costs. The methods range from training Transformer-based models on very large conversational corpora and scoring their replies with crowd-sourced human judgements, to estimating power use from hardware details, datacenter efficiency and the local electricity mix. The first strand belongs to the study of Dialogue systems.

The most cited of these papers, Towards a Human-like Open-Domain Chatbot (2020), presents Meena. Meena is a 2.6B-parameter seq2seq model built on the Evolved Transformer, and its sole training goal is predicting the next token of filtered social media conversations. The paper also proposes Sensibleness and Specificity Average (SSA), a human evaluation in which raters first check whether a reply fits its context and then whether it is specific to it. The later Carbon Emissions and Large Neural Network Training (2021) turns to the cost of this kind of work. It estimates training energy and CO2e for GPT-3, T5, Switch Transformer, GShard and Meena. It also re-examines an earlier figure for the architecture search behind the Evolved Transformer and finds that figure much too high. Taken together, the two papers move from building a large dialogue model to accounting for the resources such training consumes.

So's bylines list Google in Mountain View from 2017 to 2022, and Microsoft in Redmond as the current institution, from 2025. Co-authors on the papers here include Quoc V. Le, on two of them, along with Jeff Dean, Minh-Thang Luong, Romal Thoppilan and Noah Fiedel.

David R. So has 3 papers in our library, in 2020–2022, most of them on Dialogue systems; often with Quoc V. Le, Aakanksha Chowdhery, Apoorv Kulshreshtha.