Elamur Research⌕
☰
home / papers / 2026 / QUARTZ: Quantile-Aware Routing and Queueing for TTFT SLOs in LLM Serving

QUARTZ: Quantile-Aware Routing and Queueing for TTFT SLOs in LLM Serving

Paper · 2026 · Zhipeng Liu, Yifan Zheng, Fanqi Kong et al.

Summary

LLM serving systems often miss strict time-to-first-token (TTFT) targets set at high percentiles. The authors argue that much of this delay comes from the router queue, before any GPU work starts. Routers that ignore request size let long, cache-unfriendly prompts (for example, long RAG contexts) delay short chat requests, which is head-of-line blocking. QUARTZ replaces point estimates of request cost with conservative service-time quantiles (α = 0.9 or 0.95). It predicts them from features the router can already see: prompt length, generation budget, decoding parameters, a prefix-cache locality proxy and coarse worker backlog.

QUARTZ uses these quantiles in three ways:

  • Routing: it sends each request to the worker with the lowest predicted tail completion, chosen from a small power-of-two-choices candidate set.
  • Admission: it orders the queue by risk, with aging and token-based fairness terms.
  • Safety: it recalibrates continuously and falls back to FIFO when predictions drift.

Built into the SGLang router, it reduces P99 TTFT and SLO violations on 7B, 14B and 32B models without hurting throughput. Because it needs no engine changes, it can be deployed in existing stacks.

Key points

  • On the 14B Mixed workload, full QUARTZ reaches 3216 ms P99 TTFT and 15.8% SLO violations, compared with 3698 ms and 19.7% for the point-prediction variant (PointPred). Throughput stays at about 62k tok/s.
  • Ablations show that routing alone and queueing alone each recover only part of the gain. Calibration matters most under workload shift.
  • Calibration raises coverage at α = 0.95 from 0.819–0.887 (raw) to 0.909–0.944 across shift scenarios.
  • Fairness on 14B Mixed, compared with FIFO-LL:
    • Jain index rises from 0.919 to 0.952.
    • Maximum wait falls from 9.57 s to 7.43 s.
  • Removing the fairness terms is costly: under the Mixed Bursty workload, the Jain index drops to 0.842 and long-prompt slowdown grows to 3.07×.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Guangxi UniversityNanning · China2 works · 6 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhipeng Liu, Yifan Zheng, Fanqi Kong and Ziming Zhao. 2026. QUARTZ: Quantile-Aware Routing and Queueing for TTFT SLOs in LLM Serving. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1888

@inproceedings{liu2026quartz,
  title = {{QUARTZ: Quantile-Aware Routing and Queueing for TTFT SLOs in LLM Serving}},
  author = {Zhipeng Liu, Yifan Zheng, Fanqi Kong and Ziming Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1888},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1888}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, San… find
  2. Sohaib Ahmad, Hui Guan, and Ramesh K. Sitaraman. 2024. Loki: A system for serving ML inference pipelines with hardware and accuracy scaling. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2024, Pisa, Italy, June 3-7, 2024, pages 267–28… find
  3. Yossi Azar, Andrei Z. Broder, Anna R. Karlin, and Eli Upfal. 1999. Balanced allocations. SIAM J. Comput., 29(1):180–200. find
  4. Ida Bauer, Harry Haupt, and Stefan Linner. 2024. Pinball boosting of regression quantiles. Comput. Stat. Data Anal., 200:108027. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  6. Shiyi Cao, Yichuan Wang, Ziming Mao, Pin-Lun Hsu, Liangsheng Yin, Tian Xia, Dacheng Li, Shu Liu, Yineng Zhang, Yang Zhou, Ying Sheng, Joseph Gonzalez, and Ion Stoica. 2025. Locality-aware fair scheduling in LLM serving. CoRR, abs/2501.14312. find
  7. Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. 2020. Inferline: latency-aware provisioning and scaling for prediction serving pipelines. In SoCC ’20: ACM Symposium on Cloud Computing, Virtual Event, USA, October 19-21, 2020, pages 477–491.… find
  8. Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017, pages 613– 627. USE… find
  9. Jeffrey Dean and Luiz André Barroso. 2013. The tail at scale. Commun. ACM, 56(2):74–80. find
  10. Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient LLM scheduling by learning to rank. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15… find
  11. Mor Harchol-Balter, Bianca Schroeder, Nikhil Bansal, and Mukesh Agrawal. 2003. Size-based scheduling to improve web performance. ACM Trans. Comput. Syst., 21(2):207–233. find
  12. Mor Harchol-Balter and Ziv Scully. 2022. The most common queueing theory questions asked by computer systems practitioners. SIGMETRICS Perform. Evaluation Rev., 49(4):3–7. find
  13. Raj Jain, Dah-Ming Chiu, and W. Hawe. 1998. A quantitative measure of fairness and discrimination for resource allocation in shared computer systems. CoRR, cs.NI/9809099. find
  14. Myeongjae Jeon, Saehoon Kim, Seung-won Hwang, Yuxiong He, Sameh Elnikety, Alan L. Cox, and Scott Rixner. 2014. Predictive parallelization: taming tail latencies in web search. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, Gold Coast ,… find
  15. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz…
  16. Michael Mitzenmacher. 2001. The power of two choices in randomized load balancing. IEEE Trans. Parallel Distributed Syst., 12(10):1094–1104. find
  17. Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. 2017. Tensorflow-serving: Flexible, high-performance ML serving. CoRR, abs/1712.06139. — [arXiv:1712.06139](https://arxiv.org/abs/1712.06139) find
  18. Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar K. Iyer. 2024. Queue management for slo-oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC 2024, Redmond, WA, USA,… find
  19. Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. 2019. Conformalized quantile regression. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3538–3548. find
  20. Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. Infaas: Automated model-less inference serving. In Proceedings of the 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021, pages 397–411. USENIX Association. find
  21. Linus Schrage. 1968. Letter to the editor - A proof of the optimality of the shortest remaining processing time discipline. Oper. Res., 16(3):687–690. find
  22. Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 965–988. US… find
  23. Mohammad Siavashi, Faezeh Keshmiri Dindarloo, Dejan Kostic, and Marco Chiesa. 2025. Priority-aware preemptive scheduling for mixed-priority workloads in moe inference. In Proceedings of the 5th Workshop on Machine Learning and Systems, EuroMLSys 2025, World Trade Center, Rotterdam, The Netherlands,… find
  24. Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 173–191. USENIX A… find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, ab…
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  27. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, pages 521–538. USE… find
  28. Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2022. Enabling cost-effective, slo-aware machine learning inference serving on public cloud. IEEE Trans. Cloud Comput., 10(3):1765–1779. find
  29. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Sys…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-quartz-quantile-aware-routing-queueing/