Elamur Research⌕
☰
home / papers / 2026 / Vertical Routing: A Cost-Efficient Collaboration Routing Framework

Vertical Routing: A Cost-Efficient Collaboration Routing Framework

Paper · 2026 · Si Shen, Peijun Shen and Danhao Zhu

Summary

Systems that split work between large and small language models usually guess, before generation starts, how hard a query is and then give the whole query to one model. The authors show that lightweight difficulty predictors do barely better than chance, while stronger evaluators are costly. Their alternative, Vertical Routing, divides a single response into stages and gives the large model only the parts that matter most. Domain templates encode task structure with criticality scores, such as planning before solving in math or choosing keywords before translating. A default template has the large model write a budgeted opening prefix, after which the small model finishes. A template is chosen once per dataset using a short checklist, and large-model tokens are then allocated greedily under a cost budget. On GSM8K, MATH and WMT, the method scores slightly higher than RouterDC while using about half the tokens and showing much less run-to-run variance. This points to cheaper and more predictable LLM deployment.

Key points

  • When tested on sets split evenly between easy and hard queries in translation, math and code, lightweight query-level routers perform scarcely better than the 50% achieved by random guessing. This motivates routing at the stage level instead of the query level.
  • Against RouterDC, Vertical Routing reaches an average score of 60.4 versus 58.4. Token usage falls from 929.1 to 474.5 (−48.9%). The large model's share of output drops from 46.0% to 26.8% (41.7% relative), and standard deviation across runs is 87.5% lower.
  • A domain template is used only if three conditions hold: stages are shared, later stages depend on earlier ones, and stages clearly differ in criticality. Otherwise the prefix-first default applies. Cost is measured as weighted token counts, with the large-to-small cost ratio estimated from decoding latency.
  • On MBPP, under matched large-model budgets, the method helps most when the budget is low. The small model's arithmetic and constraint errors remain a bottleneck, and all experiments use a single model family.
  • Data and code are released at github.com/shenpeijun0212/Vertical-Routing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanjing University of Science and TechnologyNanjing · China14 works · 1,590 citations

Also on the byline: Shanghai Police College · Jiangsu Police Officer College.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Si Shen, Peijun Shen and Danhao Zhu. 2026. Vertical Routing: A Cost-Efficient Collaboration Routing Framework. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.802

@article{shen2026vertical,
  title = {{Vertical Routing: A Cost-Efficient Collaboration Routing Framework}},
  author = {Si Shen, Peijun Shen and Danhao Zhu},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.802},
  url = {https://doi.org/10.1162/tacl.a.802}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F. AbouElhamayed, Yueying Li, and Mohamed S. Abdelfattah. 2025. Splitreason: Learning to offload reasoning. find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models.
  3. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  4. Ahmadreza Jame, Marco Rapelli, and Claudio Casetti. 2022. Reducing pollutant emissions through virtual traffic lights. Computer Communications, 188:167–177. find
  5. Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. 2024a. Routerdc: Query-based router by dual contrastive learning for assembling large language models. Advances in Neural Information Processing Systems, 37:66305–66328. find
  6. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… find
  7. Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024b. Premise order matters in reasoning with large language models. find
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems.
  9. Jasper Dekoninck, Maximilian Baader, and Martin Vechev. 2025. A unified approach to routing and cascading for LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 12987–13010. PMLR. find
  10. Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Jing Li, Min Zhang, and Zhaopeng Tu. 2025. Drprun-ing: Efficient large language model pruning through distributionally robust optimization. find
  11. Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. 2024. Language model cascades: Token-level uncertainty and beyond. find
  12. Luc Bryan Heitz, Joun Chamas, and Christopher Scherb. 2024. Evaluation of the programming skills of large language models. find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding.
  14. Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. Routerbench: A benchmark for multi-llm routing system. find
  15. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations (ICLR). find
  16. Team Kimi. 2025. Kimi k1.5: Scaling reinforcement learning with llms.
  17. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovic… find
  18. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR. find
  19. Kuan-Ming Liu and Ming-Chih Lo. 2025. Llm-based routing in mixture of experts: A novel framework for trading. find
  20. Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du seong Chang, and Jiwon Seo. 2024. ExeGPT: Constraint-aware resource scheduling for LLM inference. find
  21. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. Routellm: Learning to route llms with preference data. find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computa…
  23. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… find
  24. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In International Conference on Learning Representations (ICLR).
  25. Roldao da Rocha. 2022. Holographic entanglement entropy, deformed black branes, and de-confinement in ads/qcd. Physical Review D, 105(2). find
  26. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. find
  27. Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. 2025. Dast: Difficulty-adaptive slow-thinking for large reasoning models. find
  28. Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. 2024. Drt-o1: Optimized deep reasoning translation via long chain-of-thought. arXiv e-prints, pages arXiv–2412. find
  29. Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. 2021. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. find
  30. Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. 2023. Adaptive computation with elastic input sequence. In International Conference on Machine Learning, pages 38971– 38988. PMLR. find
  31. Zheming Yang, Yuanhao Yang, Chang Zhao, Qi Guo, Wenkai He, and Wen Ji. 2024. Per-LLM: Personalized inference scheduling with edge-cloud collaboration for diverse LLM services. find
  32. Qianjin Yu, Keyu Wu, Zihan Chen, Chushu Zhang, Manlin Mei, Lingjun Huang, Fang Tan, Yongsheng Du, Kunlin Liu, and Yurui Zhu. 2025. Rethinking the generation of high-quality cot data from the perspective of llm-adaptive question difficulty grading. find
  33. Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2024. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. find
  34. Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P. Xing, Hongyi Wang, and Huaxiu Yao. 2025. CITER: Collaborative inference for efficient large language model decoding with token-level routing. find
  35. Denlu Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Carigut, William Fedus, V Le Quoc, and Denny Zhou. 2023. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations (ICLR).
  36. Richard Šléher, William Brach, Tibor Sloboda, Kristián Košt’ál, and Lukas Galke. 2025. Guarded query routing for large language models. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-vertical-routing-cost-efficient-collaboration/