Elamur Research⌕
☰
home / papers / 2026 / From Language to Driving: A Dual-Loop SLM-Enhanced Framework for Multi-Planner Scheduling via a Domain-Specific Language

From Language to Driving: A Dual-Loop SLM-Enhanced Framework for Multi-Planner Scheduling via a Domain-Specific Language

Paper · 2026 · Jiawei Liu, Xun Gong, Muli Yang et al.

Summary

The paper addresses how a robotaxi can safely carry out occasional, free-form passenger requests at the maneuver level, such as asking it to overtake or not to miss an upcoming turn. The authors treat this as choosing and sequencing several motion planners, and they build a two-tier architecture. A slow outer tier runs a small language model (SLM). It reads a textual traffic-graph description of the scene, infers what the passenger wants, and writes a schedule. A fast inner tier executes that schedule through constrained MPC with a safety fallback. Three measures make a compact model workable:

  • a receding-horizon scheme that splits long tasks into short segments;
  • SchedulingDSL, a purpose-built language that confines the model's output to scheduling logic;
  • reinforcement-learning post-training inside the nuPlan simulator.

In closed-loop nuPlan tests, a 0.5B-parameter SLM outperformed a 671B-parameter LLM baseline on instruction realization. It also kept safety, rule compliance and comfort comparable to specialized driving planners. The work matters because it shows that a small, onboard-sized model can give a traceable language-to-control chain, rather than relying on huge models or opaque end-to-end systems.

Key points

  • The framework reaches an instruction-realization score of 0.91, which is 8% above the 671B-parameter baseline OSGS. It uses only a 0.5B-parameter SLM and is evaluated closed-loop in nuPlan.
  • In the ablation, the SLM scored 0 on realization when it had to produce a full long-horizon plan in one pass in Python (a general-purpose language), with or without RL. Receding-horizon decomposition gave the best results.
  • Latency tests show realization falling as inference delay grows: 0.68 at 1s, 0.43 at 2s and 0.37 at 3s. Over the same delays, the collision metric stayed at 0.98 and TTC stayed around 0.88–0.92.
  • During RL training, reward kept rising, but late in training both response length and test realization dropped sharply. The authors attribute this to overfitting, where the model memorizes near-fixed plans.
  • The stated limitations are that the inputs are text-only, with no VLM, and that nuPlan has no ego-view rendering. All experiments were run in simulation only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSoftware · 92 works

Cite

Cite as Jiawei Liu, Xun Gong, Muli Yang, Xingrui Yu, Fen Fang, Xulei Yang, Ivor Tsang, Yunfeng hu, Hong Chen and Qing Guo. 2026. From Language to Driving: A Dual-Loop SLM-Enhanced Framework for Multi-Planner Scheduling via a Domain-Specific Language. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1381

@inproceedings{liu2026language,
  title = {{From Language to Driving: A Dual-Loop SLM-Enhanced Framework for Multi-Planner Scheduling via a Domain-Specific Language}},
  author = {Jiawei Liu, Xun Gong, Muli Yang, Xingrui Yu, Fen Fang, Xulei Yang, Ivor Tsang, Yunfeng hu, Hong Chen and Qing Guo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1381},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1381}
}

References

48 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages… find
  2. Haotian Chi, Zeyu Feng, Yueming Lyu, Chengqi Zheng, Linbo Luo, Yew-Soon Ong, Ivor Tsang, Hechang Chen, Yi Chang, and Haiyan Yin. 2025. Instructflow: Adaptive symbolic constraint-guided code generation for long-horizon planning. In The Thirty-ninth Annual Conference on Neural Information Processing S… find
  3. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2024. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine, 16(4):81–94. find
  4. Mark L Darby and Michael Nikolaou. 2012. Mpc: Current practice and challenges. Control Engineering Practice, 20(4):328–342. find
  5. Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. 2023. Parting with misconceptions about learning-based vehicle motion planning. In Proceedings of the Conference on Robot Learning, pages 1268–1281. find
  6. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning, pages 1–16. PMLR. find
  7. Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. 2025. Posterior-grpo: Rewarding reasoning processes in code generation. arXiv preprint arXiv:2508.05170. — [arXiv:2508.05170](https://arxiv.org/abs/2508.05170) find
  8. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-constrained decoding for structured NLP tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10932– 10952. find
  9. Xun Gong, Jieyu Wang, Baolin Ma, Liang Lu, Yunfeng Hu, and Hong Chen. 2021. Real-time integrated power and thermal management of connected hevs based on hierarchical model predictive control. IEEE/ASME Transactions on Mechatronics, 26(3):1271–1282. find
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and Xiao. Bi. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Petar V Kokotovic, Robert E O’Malley Jr, and Pedda-pullaiah Sannuti. 1976. Singular perturbations and order reduction in control theory—an overview. Automatica, 12(2):123–132. find
  12. Mikko Lauri, David Hsu, and Joni Pajarinen. 2023. Partially observable markov decision processes in robotics: A survey. IEEE Transactions on Robotics, 39(1):21–40. find
  13. Edouard Leurent. 2018. An environment for autonomous driving decision-making. find
  14. Xu Han, Xianda Chen, Zhenghan Cai, Pinlong Cai, Meixin Zhu, and Xiaowen Chu. 2024. From words to wheels: Automated style-customized policy generation for autonomous driving. arXiv preprint arXiv:2409.11694. — [arXiv:2409.11694](https://arxiv.org/abs/2409.11694) find
  15. Jiawei Liu, Xun Gong, Fen Fang, Muli Yang, Bohao Qu, Yunfeng Hu, Hong Chen, Xulei Yang, and Qing Guo. 2026. Open-ended instruction realization with llm-enabled multi-planner scheduling in autonomous vehicles. arXiv preprint arXiv:2604.08031. — [arXiv:2604.08031](https://arxiv.org/abs/2604.08031) find
  16. Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M Patel, and Fatih Porikli. 2025. Distilling multi-modal large language models for autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Con… find
  17. Jiawei Liu, Yanjiao Liu, Xun Gong, Tingting Wang, Hong Chen, and Yunfeng Hu. 2025. Harnessing and evaluating the intrinsic extrapolation ability of large language models for vehicle trajectory prediction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association… find
  18. Jun Hu, Yuefeng Wang, Shuai Cheng, Jinghan Xu, Ningjia Wang, Bingjie Fu, Zuotao Ning, Jingyao Li, Hualin Chen, and Chaolu Feng. 2025. A survey of decision-making and planning methods for self-driving vehicles. Frontiers in Neurorobotics, 19:1451923. find
  19. Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. 2024. Making large language models better planners with reasoning-decision alignment. In European Conference on Computer Vision, pages 73–90. find
  20. Anqing Jiang, Yu Gao, Yiru Wang, Zhigang Sun, Shuo Wang, Yuwen Heng, Hao Sun, Shichen Tang, Lijuan Zhu, and Jinhao Chai. 2025. Irl-vla: Training an vision-language-action policy via reward world model. arXiv preprint arXiv:2508.06571. — [arXiv:2508.06571](https://arxiv.org/abs/2508.06571) find
  21. Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. 2024. Senna: Bridging large vision-language models and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313. — [arXiv:2410.22313](https://arxiv.org/abs/2410.22313) find
  22. Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, and Holger Caesar. 2024. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 202… find
  23. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of rlhf on llm generalisation and diversity. In The Twelfth International Conference on Learning Representations.
  24. Yuechen Luo, Fang Li, Shaoqing Xu, Zhiyi Lai, Lei Yang, Qimao Chen, Ziang Luo, Zixun Xie, Shengyin Jiang, and Jiaxin Liu. 2025. Adathinkdrive: Adaptive thinking via reinforcement learning for autonomous driving. arXiv preprint arXiv:2509.13769. — [arXiv:2509.13769](https://arxiv.org/abs/2509.13769) find
  25. Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, and Aniket. Bera. 2024. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn… find
  26. Doaa Mahmud, Hadeel Hajmohamed, Shamma Al-mentheri, Shamma Alqaydi, Lameya Aldhaheri, Ruhul Amin Khalil, and Nasir Saeed. 2025. Integrating llms with its: Recent advances, potentials, challenges, and future directions. IEEE Transactions on Intelligent Transportation Systems, 26(5):5674– 5709. find
  27. Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. 2023. Gpt-driver: Learning to drive with gpt. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  28. Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. 2024. A language agent for autonomous driving. In First Conference on Language Modeling. find
  29. Jacob Mattingley, Yang Wang, and Stephen Boyd. 2011. Receding horizon control. IEEE Control Systems Magazine, 31(3):52–65. find
  30. Bernard Mettler and Olivier Toupet. 2005. Receding horizon trajectory planning with an environment-based cost-to-go function. In Proceedings of the 44th IEEE Conference on Decision and Control, pages 4071–4076. IEEE. find
  31. Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, and Jinyu Miao. 2025. Agentthink: A unified framework for tool-augmented chain-of-thought reasoning in vision-language models for autonomous driving. arXiv preprint arXiv:2505.15298. find
  32. Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. 2025. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 11993–12003. find
  33. Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, and Liam Paull. 2025. Poutine: Vision-language-trajectory pre-training and reinforcement learning post-training enable robust end-to-end autonomous driving. arXiv preprint arXiv:2506.11234. — [arXiv:2506.11234](https://arxiv.org/abs/2506.11234) find
  34. Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. 2025. Languagempc: Large language models as decision makers for autonomous driving. Preprint, arXiv:2310.03026. — [arXiv:2310.03026](https://arxiv.org/abs/2310.03026) find
  35. Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. 2024. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130. find
  36. Yu-Zhe Shi, Haofei Hou, Zhangqian Bi, Fanxu Meng, Xiang Wei, Lecheng Ruan, and Qining Wang. 2024. AutoDSL: Automated domain-specific language design for structural representation of procedures with constraints. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic… find
  37. Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. 2024. Driv-elm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274. find
  38. Paulo Tabuada. 2007. Event-triggered real-time scheduling of stabilizing control tasks. IEEE Transactions on Automatic control, 52(9):1680–1685. find
  39. Martin Treiber, Ansgar Hennecke, and Dirk Helbing. 2000. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805. find
  40. Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. 2025a. Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages… find
  41. Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Diamond, Yifan Ding, and Wenhao Ding. 2025b. Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv preprint arXiv:2511.00088. — [arXiv:2511.00088](https://arxiv.org/abs/2511.00088) find
  42. Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, MA Tao, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. 2024. Dilu: A knowledge-driven approach to autonomous driving with large language models. In The Twelfth International Conference on Learning Representations. find
  43. Yang Xing, Chen Lv, Dongpu Cao, and Peng Hang. 2021. Toward human-vehicle collaboration: Review and perspectives on human-centered collaborative automated driving. Transportation research part C: emerging technologies, 128:103199. find
  44. Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. 2024. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446. — [arXiv:2412.14446](https://arxiv.org/abs/2412.14446) find
  45. Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki. 2024. Diffusion-es: Gradient-free planning with diffusion for autonomous and instruction-guided driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni… find
  46. Fanzhi Zeng, Siqi Wang, Chuzhao Zhu, and Li Li. 2025a. Adrd: Llm-driven autonomous driving based on rule-based decision systems. arXiv preprint arXiv:2506.14299. — [arXiv:2506.14299](https://arxiv.org/abs/2506.14299) find
  47. Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. 2025b. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505.17685. — [arXiv:2505.17685](https://arxiv.org/abs/2505.17685) find
  48. Zewei Zhou, Tianhui Cai, Seth Z. Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. 2025. Autovla: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. Preprint, arXiv:2506.13757. — [arXiv:2506.13757](https://arxiv.org/abs/2506.13757) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-language-driving-dual-loop-slm/