📄 Generalising LLM Routing using Past Performance Retrieval: A Few-Shot Router is Sufficient

Summary

The paper asks whether LLM routing that adapts to new candidate models without retraining really needs complex architectures such as graph neural networks or learned model representations. The authors propose CONTEXTUAL ROUTER, a meta-evaluation approach. A new query is embedded and matched by cosine similarity against a small history of queries with known per-model scores. The similarities and neighbour scores are then combined through a light attention-style regressor to estimate each candidate's performance. The final model choice maximises estimated performance minus a cost term weighted by a tunable penalty λ. Because the input size depends on k rather than on the number of models, a new LLM can join by being scored on the history set. The method was tested on SPROUT, RouterBench, LiveBench, BigGenBench and EmbedLLM against UMR. It reached quality–cost trade-offs comparable to other generalisable routers. However, plain k-NN averaging did as well or better. The takeaway is that good retrieval over a small performance log is enough for practical, low-resource routing.

Key points

• Five routing benchmarks were used: SPROUT, RouterBench, LiveBench, BigGenBench and EmbedLLM. The main baselines were UMR and K-NN@40, and k = 40 was the default neighbourhood size.
• Changing the encoder made no significant difference. The authors compared potion-multilingual-128M (static, 256d), snowflake-arctic-embed-m-v2.0 (768d) and OpenAI text-embedding-3-large (3072d). The static model came close to the best, which was Snowflake's.
• In a low-data test on the unseen FusionBench, retrieval-based routers did as well with 1% of the history (n=53) as with 10% (n=535). This held for both original and newly added candidates. UMR, in contrast, degraded at 1%.
• To simulate realistic queries, 3000 samples each from RouterBench, EmbedLLM and Sprout were paraphrased with gpt-4o-mini. The resulting changes were small, with ΔPeak between −0.3 and 2.0 and ΔDTO_w between −3.0 and 1.1.
• The main practical cost is that a collection of queries, each paired with recorded scores for the candidate LLMs, must be kept available during inference, though it only needs to be very small.