Elamur Research⌕
☰
home / papers / 2026 / Conformal LLM Routing with Distribution-Free Safety Guarantees

Conformal LLM Routing with Distribution-Free Safety Guarantees

Paper · 2026 · Iqtedar Uddin and André Bauer

Summary

Routers that send easy queries to a cheap LLM and hard ones to an expensive LLM usually balance quality against cost. None of them formally limits how often the cheap model fails on the queries it receives. The authors frame this as safety-constrained routing and adapt a proactive conformal gate to text. A logistic regression over BAAI/bge-base-en-v1.5 embeddings scores each query before any model runs. Clopper-Pearson calibration on held-out data then picks the lowest threshold whose upper confidence bound on the routed-subset failure rate stays within α, with confidence 1 − δ. The approach requires only exchangeability, not a well-calibrated gate.

Experiments use RouterBench labels for Mixtral-8x7B (cheap) and GPT-4 (expensive) on GSM8K and MMLU. Across α values from 0.05 to 0.50, the conformal router stays within tolerance, while a threshold tuned on validation data exceeds the bound on GSM8K. The work introduces a guarantee-oriented evaluation axis for routing, which matters for practitioners who must certify cheap-model reliability.

Key points

  • Setup: routing is between Mixtral-8x7B-Instruct and GPT-4-1106-preview, a 24.5× per-query cost gap.
    • Data are GSM8K (7,450 queries) and MMLU (14,042 queries), split 55/15/15/15 into train, calibration, validation and test.
    • The tolerance is set to α = 0.30 on GSM8K and 0.20 on MMLU, and δ is fixed at 0.10 for both.
  • A query counts as unsafe for the cheap model only when the cheap model errs and GPT-4 answers correctly.
  • MMLU test results: coverage 0.903, violation 0.191, 87% cost savings.
  • GSM8K test results:
    • The conformal router reaches coverage 0.367, violation 0.280 and 35% savings.
    • The val-tuned baseline routes more queries (0.596) but violates at 0.317.
  • Feasibility diagnostic: a statistic C(π, α), computed for all 10 RouterBench cheap models, shows that routability depends jointly on model and task.
    • mixtral-8x7b scores 1.28 on GSM8K and 1.10 on MMLU.
    • code-llama-34b scores 2.75 on GSM8K but 10.92 on MMLU.
  • Limitations:
    • The gates are weak, with AUC 0.578 on GSM8K and 0.678 on MMLU.
    • The guarantee is marginal rather than per subgroup.
    • Results come from a single split and one model pair.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Illinois Institute of TechnologyChicago · United States12 works · 1,006 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Iqtedar Uddin and André Bauer. 2026. Conformal LLM Routing with Distribution-Free Safety Guarantees. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.70

@inproceedings{uddin2026conformal,
  title = {{Conformal LLM Routing with Distribution-Free Safety Guarantees}},
  author = {Iqtedar Uddin and André Bauer},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.70},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.70}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. 2022. Learn then test: Calibrating predictive algorithms to achieve risk control. Preprint, arXiv:2110.01052. — [arXiv:2110.01052](https://arxiv.org/abs/2110.01052) find
  2. Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2025. Conformal risk control. arXiv preprint arXiv:2208.02814. — [arXiv:2208.02814](https://arxiv.org/abs/2208.02814) find
  3. Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. 2021. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6):1–34. find
  4. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  5. Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. 2024. RouterDC: Query-based router by dual contrastive learning for assembling large language models. arXiv preprint arXiv:2409.19886. — [arXiv:2409.19886](https://arxiv.org/abs/2409.19886) find
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  7. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed H Awadallah. 2024. Hybrid LLM: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618. find
  8. Isaac Gibbs and Emmanuel Candès. 2021. Adaptive conformal inference under distribution shift. Preprint, arXiv:2106.00170. — [arXiv:2106.00170](https://arxiv.org/abs/2106.00170) find
  9. Sai Hao, Hao Zeng, Hongxin Wei, and Bingyi Jing. 2026. RACER: Risk-aware calibrated efficient routing for large language models. arXiv preprint arXiv:2603.06616. — [arXiv:2603.06616](https://arxiv.org/abs/2603.06616) find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  11. Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. RouterBench: A benchmark for multi-LLM routing system. arXiv preprint arXiv:2403.12031. — [arXiv:2403.12031](https://arxiv.org/abs/2403.12031) find
  12. Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, and Shuyue Hu. 2026. LLMRouterBench: A massive benchmark and unified framework for LLM routing. arXiv preprint arXiv:2601.07206. find
  13. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to route LLMs with preference data. arXiv preprint arXiv:2406.18665. find
  14. Tal Shnitzer, Anthony Ou, Mı́rian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789.
  15. Jiayuan Su, Fulin Lin, Zhaopeng Feng, Han Zheng, Teng Wang, Zhenyu Xiao, Xinlong Zhao, Zuozhu Liu, Lu Cheng, and Hongwei Wang. 2025. CP-Router: An uncertainty-aware router between LLM and LRM. arXiv preprint arXiv:2505.19970. — [arXiv:2505.19970](https://arxiv.org/abs/2505.19970) find
  16. Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. 2024. API is enough: Conformal prediction for large language models without logit-access. In Findings of EMNLP. find
  17. Iqtedar Uddin, Mazin Khider, and André Bauer. 2026. Proactive routing to interpretable surrogates with distribution-free safety guarantees. arXiv preprint arXiv:2603.14623. — [arXiv:2603.14623](https://arxiv.org/abs/2603.14623) find
  18. Antonios Valkanas, Soumyasundar Pal, Pavel Rumiantsev, Yingxue Zhang, and Mark Coates. 2025. C3po: Optimized large language model cascades with probabilistic cost constraints for reasoning. arXiv preprint arXiv:2511.07396. — [arXiv:2511.07396](https://arxiv.org/abs/2511.07396) find
  19. Vladimir Vovk, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer. find
  20. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uddin-2026-conformal-llm-routing-distribution-free/