Elamur Research⌕
☰
home / papers / 2026 / LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models

LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models

Paper · 2026 · Shivam Ratnakar, Abhiroop Talasila and Vinayak K Doifode

Summary

In a multi-agent system with about a hundred agents, choosing the right agent for each request becomes a bottleneck. Asking an LLM to pick is slow (roughly 1.5–2 s) and loses accuracy as the agent list grows. Cosine matching over embeddings is fast but confuses agents whose descriptions share vocabulary. The authors trace this weakness to anisotropy: hidden-state vectors crowd into a narrow cone, so similarity scores barely differ between classes. LatentGate keeps a small language model frozen and averages token states from an early-to-middle layer. It then standardizes these features, PCA-whitens them down to 16 dimensions and fits a logistic-regression classifier. The enterprise benchmark has 100 agents and 5,000 training queries generated by GPT-4.1 and checked by humans. Across five SLM backbones, LatentGate reaches 98.8% in-domain and 80.0% OOD accuracy in about 28 ms on a T4 GPU. This beats embedding baselines by 13–22 points and matches or exceeds prompted GPT-4.1, GPT-4.1-nano and Gemini 2.5 Flash. The work suggests that fixing representation geometry, rather than scaling models, makes cheap and fast routing feasible.

Key points

  • Diagnosis on Qwen-2.5-1.5B-Instruct (layer 7): the raw intra- vs. inter-class cosine gap is only 0.005. Whitening raises it to 0.255 (48×), while BGE-M3 reaches 0.13. In raw states, the top 3 principal components carry about 30% of the variance.
  • Real-world check on CLINC150: accuracy is 92.9% with a frozen 1.5B backbone and about 38K trainable parameters, and whitening enlarges the gap 278×.
  • Ablation: dropping PCA-whitening cuts OOD accuracy from 80.0% to 62.8%. Mean pooling beats [CLS], last-token and max pooling, and k = 16 is the best trade-off. A two-layer MLP probe adds less than 1%.
  • Scaling and cost: as agent count rises, GPT-4.1 falls from 96.0% to 77.0%, while LatentGate falls from 97.0% to 80.0%. LatentGate is about 74× faster and costs roughly $2.72 per 1M queries per month (99.98% cheaper). The probe is about 4 KB and can be refit in under 10 ms.
  • Limitations: the evaluation covers only single-turn English queries, and training and OOD data are LLM-generated with human verification (κ = 0.91). There is no sustained production deployment yet. Code and the benchmark skeleton are available at github.com/shivoxaham/latentgate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States175 works · 37,047 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksInterpretability · 149 works

Cite

Cite as Shivam Ratnakar, Abhiroop Talasila and Vinayak K Doifode. 2026. LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.153

@inproceedings{ratnakar2026latentgate,
  title = {{LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models}},
  author = {Shivam Ratnakar, Abhiroop Talasila and Vinayak K Doifode},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.153},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.153}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. 2025. Automix: Automatically mixing language models. Preprint, arXiv:2310.12963. — [arXiv:2310.12963](https://arxiv.org/abs/2310.12963) find
  2. Aurelio AI. 2024. Semantic router. //github.com/aurelio-labs/ semantic-router. https: find
  3. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2025. M3-embedding: Multilinguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  5. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. Preprint, arXiv:2305.05176.
  6. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. Preprint, arXiv:2404.14618.
  7. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  8. Ali Forooghi, Shaghayegh Sadeghi, and Jianguo Lu. 2024. Whitening not recommended for classification tasks in LLMs. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 285–289, Bangkok, Thailand. Association for Computational Linguistics. find
  9. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations.
  10. Nathan Godey, Éric de la Clergerie, and Benoît Sagot. 2024. Anisotropy is inherent to self-attention in transformers. Preprint, arXiv:2401.12143. find
  11. Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. Routerbench: A benchmark for multi-llm routing system. Preprint, arXiv:2403.12031. — [arXiv:2403.12031](https://arxiv.org/abs/2403.12031) find
  12. Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021. Whiteningbert: An easy unsupervised sentence embedding approach. Preprint, arXiv:2104.01767. find
  13. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Confer… find
  14. Elias Lumer, Faheem Nizar, Anmol Gulati, Pradeep Honaganahalli Basavaraju, and Vamse Kumar Subbiah. 2025. Tool-to-agent retrieval: Bridging tools and agents for scalable llm multi-agent systems. Preprint, arXiv:2511.01854. — [arXiv:2511.01854](https://arxiv.org/abs/2511.01854) find
  15. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. Routellm: Learning to route llms with preference data. Preprint, arXiv:2406.18665.
  16. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. Preprint, arXiv:2103.15316. find
  17. Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. Preprint, arXiv:2209.11055. — [arXiv:2209.11055](https://arxiv.org/abs/2209.11055) find
  18. Zhiyuan Yao, Zishan Xu, Yifu Guo, Zhiguang Han, Cheng Yang, Shuo Zhang, Weinan Zhang, Xingshan Zeng, and Weiwen Liu. 2026. Toolace-mcp: Generalizing history-aware routing from mcp tools to the agent web. Preprint, arXiv:2601.08276. — [arXiv:2601.08276](https://arxiv.org/abs/2601.08276) find
  19. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to route LLMs for multi-agent systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15549–15572, Vienna, Au…
  20. Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, and Yi Yang. 2023. WhitenedCSE: Whitening-based contrastive learning of sentence embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12135–12148, Toronto, Canada. Asso… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ratnakar-2026-latentgate-low-latency-semantic-routing/