Elamur Research⌕
☰
home / papers / 2026 / RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment

RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment

Paper · 2026 · Yingfeng Luo, Hongyu Liu, DingYang Lin et al.

Summary

Serving every translation request with a large LLM is too costly for industrial systems. Hybrid setups therefore send most traffic to a small model and escalate only some inputs to a large one. The authors treat this escalation decision as spending a fixed budget of large-model calls. They argue that requests should be ranked by marginal gain, meaning how much better the large model scores than the small one. Input difficulty and the small model's absolute quality are, in their view, the wrong signals.

RouteLMT builds this predictor into the small translator itself. After a single prefill pass, a linear regression head reads the hidden state of the final prompt token. The head is trained together with LoRA adapters using an MSE loss against gain labels. The router needs no external QE model and no draft translation, and the prompt makes it direction-aware.

On En↔Zh and En↔Ru, gain-based routing achieves a better quality–budget Pareto frontier than heuristics and quality/difficulty predictors. This offers a cheap, low-latency way to reduce deployment costs.

Key points

  • Data and evaluation: Training uses the General Translation split of ComMT. Evaluation combines FLORES-200 devtest, WMT24++ and BOUQuET. Four directions are covered: En↔Zh and En↔Ru.
  • Model setup: The hybrid pairs a 0.6B small model with an 8B large model. Gain labels are differences in reference-based quality scores, with case-study scores computed by XCOMET-XXL. The authors note that a reference-free QE metric could replace the reference-based one.
  • Guarded routing at a 30% budget (p=0.3):
    • Pure gain ranking raises mean improvement over random routing from 5.83 to 12.13.
    • Its severe-loss rate is similar to random (8.19% vs 7.10%).
    • Adding a post-decoding quality check on hypotheses lowers severe losses to 5.69% and lifts mean gain to 16.73, at the cost of extra decoding.
  • Sources of regressions: Severe losses mostly come from rare large-model errors, such as resolving "WA" to the wrong entity or paraphrases that drift in meaning. Because these errors are unrelated to input difficulty, severe-loss rates look alike across routing policies.
  • Limitations: Supervision depends on an automatic metric. Only one two-model pair and four directions are tested. Multi-tier cascades and latency-aware objectives are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityShenyang · China53 works · 2,205 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yingfeng Luo, Hongyu Liu, DingYang Lin, Kaiyan Chang, Chenglong Wang, Bei Li, Quan Du, Tong Xiao and JingBo Zhu. 2026. RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.129

@inproceedings{luo2026routelmt,
  title = {{RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment}},
  author = {Yingfeng Luo, Hongyu Liu, DingYang Lin, Kaiyan Chang, Chenglong Wang, Bei Li, Quan Du, Tong Xiao and JingBo Zhu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.129},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.129}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duarte M. Alves, José Pombal, Nuno Miguel Guerreiro, Pedro Henrique Martins, João Alves, M. Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-r…
  2. Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R Costa-jussà, Joe Chuang, David Dale, Mark Duppenthaler, Nathanial Paul Ekberg, Cynthia Gao, Daniel Edward Licht, and 1 others. 2025. Bouquet: dataset, benchmark and open initiative for universal quality evaluation in translation. In Proc… find
  3. Lingjiao Chen, Matei Zaharia, and James Zou. 2024. Frugalgpt: How to use large language models while reducing cost and improving performance.
  4. Trans. Mach. Learn. Res., 2024. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  6. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, an… find
  7. Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, and Bin Wang. 2025. Multilingual machine translation with open large language models at practical scale: An empirical study. In NAACL (Long Papers), pages 5420–5443. Association for Computational Linguistics. find
  8. Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. WMT24++: Expanding the Lang…
  9. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria,…
  10. Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2022. Prequel: Quality estimation of machine translation outputs in advance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11170–11… find
  11. Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. 2025. Emergent response planning in llms. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. find
  12. António Farinhas, Nuno M Guerreiro, Sweta Agrawal, Ricardo Rei, and André FT Martins. 2025. Translate smart, not hard: Cascaded translation systems with quality-aware deferral. arXiv preprint arXiv:2502.12701. find
  13. Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.
  14. Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. 2024. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  15. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. find
  16. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  17. Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. CoRR, abs/2305.18098. — [arXiv:2305.18098](https://arxiv.org/abs/2305.18098) find
  18. Yubo Zhu, Dongrui Liu, Zecheng Lin, Wei Tong, Sheng Zhong, and Jing Shao. 2025. The llm already knows: Estimating llm-perceived question difficulty via hidden representations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1160–1176. find
  19. Sunbowen Lee, Qingyu Yin, Chak Tou Leong, Jialiang Zhang, Yicheng Gong, Shiwen Ni, Min Yang, and Xiaoyu Shen. 2025. Probing the difficulty perception mechanism of large language models. arXiv preprint arXiv:2510.05969. — [arXiv:2510.05969](https://arxiv.org/abs/2510.05969) find
  20. Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang, Murun Yang, Dingyang Lin, Kaiyan Chang, Tong Zheng, Bei Li, Peinan Feng, Quan Du, and 1 others. 2025a. Beyond english: Toward inclusive and scalable multilingual machine translation with llms. arXiv preprint arXiv:2511.07003. — [arXiv:2511.07003](https://arxiv.org/abs/2511.07003) find
  21. Yingfeng Luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, and JingBo Zhu. 2025b. Beyond decoder-only: Large language models can be good encoders for machine translation. In Findings of the Association for Computational Linguistics, ACL… find
  22. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. CoRR, abs/2406.18665.
  23. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating machine translation difficulty. Preprint. find
  24. Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. 2025. Mixllm: Dynamic routing in mixed large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hu… find
  25. Zhanglin Wu, Daimeng Wei, Xiaoyu Chen, Hengchao Shang, Jiaxin Guo, Zongyao Li, Yuanchang Luo, Jinlong Yang, Zhiqiang Rao, and Hao Yang. 2025. Combining the best of both worlds: A method for hybrid nmt and llm translation. In Annual Meeting of the Association for Computational Linguistics. find
  26. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. A paradigm shift in machine translation: Boosting translation performance of large language models. In ICLR. OpenReview.net. find
  27. A Hyperparameter Train Type Learning Rate Number of Epochs Global Batch Size Max Length Warmup Ratio LoRA Rank LoRA Alpha Target Modules Precision Baseline Details find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-routelmt-learned-sample-routing-hybrid/