Elamur Research⌕
☰
home / papers / 2026 / GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

Paper · 2026 · Wenhao Zeng, Xuteng Zhang, Yuling Shi et al.

Summary

Large reasoning models produce long chains of thought, which makes inference slow and expensive. Collaborative inference addresses this by splitting work between a small and a large model. The hard part is deciding cheaply which model should handle each step. Existing step-level routers either generate a full step and then score it (with averaged entropy or perplexity, or LLM-as-a-judge), or switch models token by token. Both add overhead.

The authors observe that a step's difficulty shows up in its opening token, echoing the "Aha Moment" pivots marked by words like "Wait" or "So". They propose GlimpRouter, which needs no training. A small model drafts only the first token of each step. If that token's entropy is low, the small model writes the whole step. If it is high, the large model takes over. This "Probe-then-Dispatch" design avoids discarded drafts.

On AIME, GPQA and LiveCodeBench, the method gives a better accuracy–latency trade-off than running the large model alone. It also stacks with speculative decoding for further speedups.

Key points

  • Preliminary study: The authors analysed reasoning traces from Qwen3-4B, Qwen3-32B and DeepSeek-R1-Distill-Qwen-32B on AIME and LiveCodeBench, totalling over 10M tokens.
    • First-token entropy has a bimodal, heavy-tailed distribution.
    • Step-wise entropy and perplexity cluster narrowly around central values.
    • LLM-as-a-judge scores pile up near the top of the scale.
  • Headline result: On AIME25, GlimpRouter raises accuracy by 10.7% and cuts latency by 25.9% relative to the standalone large model.
  • Ablation on AIME25: Routing by initial-token entropy reached 51.67 accuracy at 163s. Step-wise entropy gave 46.67 at 178s, and step-wise perplexity gave 47.50 at 181s.
  • With speculative decoding: Adding it lowers latency further.
    • On AIME25: 130s at 51.67 accuracy, versus 220s at 46.67 for the large model alone.
    • On LCBv6: 137s, the fastest configuration tested, though at 48.00 versus 48.29 without speculative decoding.
    • Both settings also beat the SpecReason baseline.
  • Limitations: The method uses a fixed global entropy threshold. It also splits steps at double newlines, which may not suit models that produce unstructured reasoning. Code and data are available on GitHub (Zengwh02/GlimpRouter).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Wenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu, Yuting Chen, Beijun Shen and Xiaodong Gu. 2026. GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.885

@inproceedings{zeng2026glimprouter,
  title = {{GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts}},
  author = {Wenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu, Yuting Chen, Beijun Shen and Xiaodong Gu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.885},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.885}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.
  2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023a. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  3. Lingjiao Chen, Matei Zaharia, and James Zou. 2023b. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  4. Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. 2025a. Swe-exp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361. — [arXiv:2507.23361](https://arxiv.org/abs/2507.23361) find
  5. Yi Chen, JiaHao Zhao, and HaoHao Han. 2025b. A survey on collaborative mechanisms between large and small language models. arXiv preprint arXiv:2505.07460. find
  6. Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, and 1 others. 2025. Stepwise perplexity-guided refinement for efficient chain-of-thought reasoning in large language models. arXiv preprint arXiv:2502.13260. find
  7. Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. 2024. Learning how hard to think: Input-adaptive allocation of lm computation. arXiv preprint arXiv:2410.04707. find
  8. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618. find
  9. Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. 2025. R2r: Efficiently navigating divergent reasoning paths with small-large model token routing. arXiv preprint arXiv:2505.21600. — [arXiv:2505.21600](https://arxiv.org/abs/2505.21600) find
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Minghao Hu, Junzhe Wang, Weisen Zhao, Qiang Zeng, and Lannan Luo. 2025. Flowmaltrans: Unsupervised binary code translation for malware detection using flow-adapter architecture. arXiv preprint arXiv:2508.20212. find
  12. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  13. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  14. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. find
  15. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  16. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. find
  17. Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. 2025. Swe-debate: Competitive multi-agent debate for software issue resolution. arXiv preprint arXiv:2507.23348. — [arXiv:2507.23348](https://arxiv.org/abs/2507.23348) find
  18. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077.
  19. Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. 2025. Reward-guided speculative decoding for efficient llm reasoning. In Forty-second International Conference on Machine Learning. find
  20. Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507.
  21. Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650. find
  22. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665. find
  23. OpenAI. 2025. Openai o3 and o4-mini system card. find
  24. Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. 2025. Specreason: Fast and accurate inference-time compute via speculative reasoning. arXiv preprint arXiv:2504.07891. — [arXiv:2504.07891](https://arxiv.org/abs/2504.07891) find
  25. Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, and 1 others. 2025. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614.
  26. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  27. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  28. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  29. Jianshu She, Zhuohao Li, Zhemin Huang, Qi Li, Peiran Xu, Haonan Li, and Qirong Ho. 2025. Hawkeye: Model collaboration for efficient reasoning. In Second Conference on Language Modeling. find
  30. Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, and Yong Jiang. 2025. Speccot: Accelerating chain-of-thought reasoning through speculative exploration. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24405– 24415. find
  31. Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025a. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502.18017.
  32. Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, and Feng Zhao. 2025b. Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning. arXiv preprint arXiv:2505.22019.
  33. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025c. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939.
  34. Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. 2025d. Mixllm: Dynamic routing in mixed large language models. arXiv preprint arXiv:2502.18482. find
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  36. Zilin Xiao, Jaywon Koo, Siru Ouyang, Jefferson Hernandez, Yu Meng, and Vicente Ordonez. 2025. Prox-ythinker: Test-time guidance through small visual reasoners. arXiv preprint arXiv:2505.24872. — [arXiv:2505.24872](https://arxiv.org/abs/2505.24872) find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  38. Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. 2025b. Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time. arXiv preprint arXiv:2504.12329. — [arXiv:2504.12329](https://arxiv.org/abs/2504.12329) find
  39. Xinwei Yang, Zhaofeng Liu, Chen Huang, Jiashuai Zhang, Tong Zhang, Yifan Zhang, and Wenqiang Lei. 2025c. Elaboration: A comprehensive benchmark on human-llm competitive programming. arXiv preprint arXiv:2505.16667. find
  40. Qingcheng Zeng, Weihao Xuan, Leyang Cui, and Rob Voigt. 2025a. Thinking out loud: Do reasoning models know when they’re right? arXiv preprint arXiv:2504.06564.
  41. Wenhao Zeng, Yaoning Wang, Chao Hu, Yuling Shi, Chengcheng Wan, Hongyu Zhang, and Xiaodong Gu. 2025b. Pruning the unsurprising: Efficient code reasoning via first-token surprisal. arXiv preprint arXiv:2508.05988. — [arXiv:2508.05988](https://arxiv.org/abs/2508.05988) find
  42. Jinghan Zhang, Xiting Wang, Fengran Mo, Yeyang Zhou, Wanfu Gao, and Kunpeng Liu. 2025a. Entropy-based exploration conduction for multi-step reasoning. arXiv preprint arXiv:2503.15848. find
  43. Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, and Keze Wang. 2025b. CF-VLM: Counterfactual vision-language fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS). find
  44. Jusheng Zhang, Kaitong Cai, Xiaoyang Guo, Sidi Liu, Qinhan Lv, Ruiqi Chen, Jing Yang, Yijia Fan, Xiaofei Sun, Jian Wang, Chen Ziliang, Liang Lin, and Keze Wang. 2025c. Mm-cot: A benchmark for probing visual chain-of-thought reasoning in multimodal models. Preprint, arXiv:2512.08228. — [arXiv:2512.08228](https://arxiv.org/abs/2512.08228) find
  45. Jusheng Zhang, Yijia Fan, Wenjun Lin, Ruiqi Chen, Haoyi Jiang, Wenhao Chai, Jian Wang, and Keze Wang. 2025d. GAM-agent: Game-theoretic and uncertainty-aware collaboration for complex visual reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS). find
  46. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557– 62…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-glimprouter-efficient-collaborative-inference-glimpsing/