🧑‍🔬 Cody Hao Yu

Cody Hao Yu works on systems for running large language models efficiently. Their papers here, both from 2023, focus on making inference faster and less wasteful. The work examines how GPU memory is spent on the key-value cache and how applications that chain many generation calls can be executed with less redundant computation. The approach borrows from operating systems and programming languages, pairing new abstractions with runtime designs. These papers fall under Language models & pretraining and Efficient models & compression.

The most cited of these papers is Efficient Memory Management for Large Language Model Serving with PagedAttention. It starts from the observation that serving systems set aside a large contiguous block of memory for every request. Because cache size varies unpredictably, much of that space is lost to fragmentation, and shared portions end up stored more than once. The paper adapts paged virtual memory to this setting, so the cache can be held in non-contiguous blocks and batches can grow larger. The second paper, SGLang: Efficient Execution of Structured Language Model Programs, targets "LM programs" that combine branching logic with output in fixed formats. Its frontend is a small language embedded in Python, offering operations for generating text, choosing among options, and splitting work into parallel branches and merging it back. A runtime then keeps cached state between calls instead of discarding it and speeds up constrained decoding.

Both papers are also the newest here. Read together, they span low-level memory management for serving and end-to-end support for structured LLM workloads. Yu's collaborators on this work include Lianmin Zheng, Ying Sheng, Ion Stoica and Joseph E. Gonzalez, each of whom appears on both papers.

Cody Hao Yu has 2 papers in our library, in 2023, most of them on Language models & pretraining, Efficient models & compression; often with Ion Stoica, Joseph E. Gonzalez, Lianmin Zheng.