Elamur Research⌕
☰
home / papers / 2026 / Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching

Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching

Paper · 2026 · Bo Lv, Jingbo Sun, Jianwei Lv et al.

Summary

The paper addresses LLM routing: picking, for each incoming query, a model from a pool that is good enough while keeping inference cost low. The authors argue that current routers learn a direct query-to-model mapping from surface features. This makes them prone to a "memorization trap" and weak on out-of-distribution (OOD) inputs. Such routers also need retraining whenever the model pool changes.

Their framework, DecoR, instead treats routing as matching against historical query–response logs. A Query Deconstructor first converts each query into a capability profile of Skills, Knowledge and Difficulty. Logs with similar profiles are then filtered in three tiers. Finally, a model is chosen by weighing its past performance against cost on those matched logs. If nothing matches, a strong default model is used as a fallback.

The authors also release CodaSet, a benchmark built with frontier models that has in-distribution (ID) and OOD splits. DecoR achieves higher accuracy at lower cost than existing routers in both settings. Because the system can be updated simply by adding logs, it avoids retraining, which makes it practical for evolving model pools.

Key points

  • Capability profile:
    • Skill and Knowledge categories are generated freely per query rather than taken from a fixed list.
    • Difficulty uses four levels (d0 trivial to d3 deep reasoning).
    • Each dimension comes with a short justification.
  • First sifting stage:
    • Jaccard overlap on skills and knowledge, looked up through an inverted index, with a threshold τ (default 0.5).
    • Logs below the threshold mark the query as OOD and trigger the fallback model.
    • Removing this stage reduces GSM8k accuracy from 95.54 to 91.70 and BBH from 93.89 to 88.57.
  • Setup:
    • LLM pool of 8 models: Kimi-K2-Instruct, DeepSeek-V3.1, DeepSeek-V3.2-Exp, Qwen3-235B, gpt-oss-120b, gemma-3-27b-it, Mistral-3.2-24B and gemma-3-12b-it.
    • DecoR is compared against six baseline routers: MODEL-SAT, EmbedLLM, RouterDC, a KNN-based router, an LLM-based router and a random router.
  • Benchmark splits: the CodaSet ID split covers GSM8k, MMLU-PRO and BBH; the OOD split covers Math_500, MBPP and MT_Bench. With DeepSeek-V3.1 as fallback, DecoR scores 95.59, 80.36 and 93.89 on the three ID sets.
  • Model size: using small Qwen3 models (0.6b–4.0b) as the Deconstructor and Evaluator changes results very little (ID average ≈90, OOD average ≈87). Code and data are available at github.com/lvbotenbest/DecoR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓Tianjin UniversityTianjin · China41 works · 1,128 citations🔬Peng Cheng LaboratoryShenzhen · China38 works · 2,504 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Bo Lv, Jingbo Sun, Jianwei Lv, Chen Tang, Shaojie Zhang, Nayu Liu, Guoxin Yu, Zihao Li, Qichao Zhang, Dongbin Zhao, Ping Luo and Yue Yu. 2026. Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1852

@inproceedings{lv2026beyond,
  title = {{Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching}},
  author = {Bo Lv, Jingbo Sun, Jianwei Lv, Chen Tang, Shaojie Zhang, Nayu Liu, Guoxin Yu, Zihao Li, Qichao Zhang, Dongbin Zhao, Ping Luo and Yue Yu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1852},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1852}
}

References

24 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  2. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  3. Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. 2024b. RouterDC: Query-based router by dual contrastive learning for assembling large language models. In Neural Information Processing Systems. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  6. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the frontier of open…
  7. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations. find
  8. Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv: 2403.12031. — [arXiv:2403.12031](https://arxiv.org/abs/2403.12031) find
  9. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  10. Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find
  11. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. Preprint, arXiv:2406.18665. find
  12. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  13. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  14. Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789.
  15. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  16. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  17. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025b. Kimi k2: Open agentic intelligence. Preprint,…
  18. Chenxu Wang, Hao Li, Yiqun Zhang, Linyao Chen, Jianhao Chen, Ping Jian, Peng Ye, Qiaosheng Zhang, and Shuyue Hu. 2025. Icl-router: In-context learned model representations for llm routing. Preprint, arXiv:2510.09719. — [arXiv:2510.09719](https://arxiv.org/abs/2510.09719) find
  19. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574.
  20. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  21. Yi-Kai Zhang, De-Chuan Zhan, and Han-Jia Ye. 2025. Capability instruction tuning: A new paradigm for dynamic llm routing. Preprint, arXiv:2502.17282. — [arXiv:2502.17282](https://arxiv.org/abs/2502.17282) find
  22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3…
  23. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
  24. Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. 2025. Em-bedLLM: Learning compact representations of large language models. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lv-2026-beyond-query-memorization-large-language/