🧑‍🔬 Daniel Fu

Daniel Fu works on large language models from the systems side and the data side. Their papers here address two practical problems: how to run very large models on modest hardware, and how to give open research access to pretraining corpora that can be inspected and reproduced. The work sits within Language models & pretraining and GPU & parallel computing. It combines engineering of memory and computation with the building and documentation of large public datasets.

The most cited of these papers is FlexGen (2023). It presents an inference engine aimed at batch workloads, where response time matters less than total throughput. FlexGen treats GPU memory, CPU memory and disk as a single storage hierarchy. It uses linear programming to decide where model weights, activations and the attention cache should sit and when they should move, and it compresses parts of the model to 4 bits. This lets a single commodity GPU serve models that would normally need several accelerators. The second paper, RedPajama (2024), answers the lack of transparency around how leading models are trained. Its first release reconstructs the LLaMA-1 training mixture and records each ambiguous choice made along the way. Its second release offers a very large, lightly filtered Common Crawl corpus that comes with per-document quality signals, so users can build their own filtering.

The newer of the two works moves from efficient inference toward open pretraining data. Fu is currently affiliated with the University of California San Diego and with Together, and earlier papers carry the Stanford University byline. Co-authors on these papers include Ying Sheng, Christopher Re and Percy Liang.

Daniel Fu has 2 papers in our library, in 2023–2024, most of them on Language models & pretraining, GPU & parallel computing; often with Anton Alexandrov, Beidi Chen, Ben Athiwaratkun.