Elamur Research⌕
☰
home / papers / 2026 / Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents

Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents

Paper · 2026 · Tianyi Ma, Yue Zhang, Zehao Wang et al.

Summary

Vision-and-Language Navigation (VLN) agents still generalize poorly to unseen buildings and unfamiliar instruction phrasing. End-to-end models tend to memorize training trajectories. Zero-shot LLM agents are more stable but fall well behind fine-tuned systems because their actions are weakly grounded. SkillNav addresses this by splitting navigation into reusable atomic skills, such as changing direction, moving between floors, recognizing rooms, spotting landmarks and deciding when to stop. Each skill has its own expert agent, fine-tuned from the DUET graph-transformer backbone on synthetic instruction–trajectory pairs generated automatically, with no human annotation. A training-free VLM router coordinates these experts. It reorders the instruction into a temporally correct list of subgoals, works out which subgoal is currently active from the agent's observations and past actions, and then hands control to the best-suited expert. The system performs competitively on R2R and reaches state-of-the-art generalization on GSA-R2R, which features dialogue-style instructions and unseen residential and non-residential scenes. The approach also makes decisions more interpretable and offers a better efficiency–generalization balance than LLM-heavy agents.

Key points

  • The skill taxonomy extends NavNuances. Five specialized agents result from it, one each for Landmark Detection, Area and Region Identification, Stop and Pause, Vertical Movement, and Direction Adjustment. Temporal ordering is handled by the router's Temporal Reordering Module.
  • The router has three stages: Temporal Reordering Module, Subgoal Localizer and Skill Router. It activates a single expert per subgoal instead of running an ensemble at every step.
  • The authors report a gap of roughly 36% in Success Rate between zero-shot LLM agents and fine-tuned VLN models, which motivates the hybrid design.
  • Training cost is about 55.5 hours, compared with about 42 hours for SRDF training on augmented R2R. This is a one-time cost, and no retraining is needed for new domains.
  • On GSA-R2R Test-N-Basic, inference throughput is 0.49 inferences/s, which is about 50× slower than ScaleVLN but about 20× faster than MapGPT. With Qwen2.5-VL-7B, the router takes 1.57 s per step. Total latency is 9.69 s per episode, 2–4× faster than NavGPT and FlexVLN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations🎓Michigan State UniversityEast Lansing · United States23 works · 1,077 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksAutomotive Engineering · 6 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Tianyi Ma, Yue Zhang, Zehao Wang and Parisa Kordjamshidi. 2026. Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.595

@inproceedings{ma2026breaking,
  title = {{Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents}},
  author = {Tianyi Ma, Yue Zhang, Zehao Wang and Parisa Kordjamshidi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.595},
  url = {https://doi.org/10.18653/v1/2026.acl-long.595}
}

References

48 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. 2023. Bevbert: Multimodal map pre-training for language-guided navigation. Proceedings of the IEEE/CVF International Conference on Computer Vision. find
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer visi…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  4. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV). find
  5. Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K. Wong. 2024a. MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation. arXiv preprint. ArXiv:2401.07314 [cs]. find
  6. Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H. Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. 2023. A2nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models. CoRR, abs/2308.07997. — [arXiv:2308.07997](https://arxiv.org/abs/2308.07997) find
  7. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. Advances in neural information processing systems, 34:5834–5847. find
  8. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022a. Learning from unlabeled 3d environments for vision-and-language navigation. In ECCV. find
  9. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022b. Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation. arXiv preprint. ArXiv:2202.11742 [cs]. find
  10. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and P…
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  12. Vision-language navigation with random environmental mixup. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1644–1654. find
  13. Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13137–13146. find
  14. Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. 2024. Discuss before moving: Visual language navigation via multi-expert discussions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17380–17387. IEEE. find
  15. Haodong Hong, Yanyuan Qiao, Sen Wang, Jiajun Liu, and Qi Wu. 2025. General scene adaptation for vision-and-language navigation. In The Thirteenth International Conference on Learning Representations. find
  16. Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. 2021. A Recurrent Vision-and-Language BERT for Navigation. arXiv preprint. ArXiv:2011.13922 [cs]. — [arXiv:2011.13922](https://arxiv.org/abs/2011.13922) find
  17. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, and 17 others. 2… find
  18. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding. arXiv preprint. ArXiv:2010.07954. find
  19. Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, and 1 others. 2024. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision, 16(1-2):1–214. find
  20. Jialu Li and Mohit Bansal. 2023. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS). find
  21. Jialu Li, Hao Tan, and Mohit Bansal. 2022. Envedit: Environment editing for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15407–15417. find
  22. Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. 2024. NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning. find
  23. Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. 2021. find
  24. OpenAI. 2024. Hello gpt-4o. find
  25. Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton Van Den Hengel. 2020. REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9979– 9988, Seattle, WA… find
  26. Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu. 2023. March in chat: Interactive prompting for remote embodied referring expression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15758–15767. find
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  28. Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. 2021. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments f… find
  29. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  30. Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. arXiv preprint arXiv:1904.04195.
  31. Liuyi Wang, Zongtao He, Jinlong Li, Xiaoyan Qi, Mengxian Hu, Chenpeng Yao, Chengju Liu, and Qijun Chen. 2025a. Clash: Collaborative large-small hierarchical framework for continuous vision-and-language navigation. Preprint, arXiv:2512.10360. — [arXiv:2512.10360](https://arxiv.org/abs/2512.10360) find
  32. Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, and Tinne Tuytelaars. 2024. Navigating the nuances: A fine-grained evaluation of vision-language navigation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4681– 4704, Miami, Florida, USA. Association for Computat… find
  33. Chongyang Zhao, Yuankai Qi, and Qi Wu. 2023. Mind the Gap: Improving Success Rate of Vision-and-Language Navigation by Revisiting Oracle Success Routes. arXiv preprint. ArXiv:2308.03244 [cs]. — [arXiv:2308.03244](https://arxiv.org/abs/2308.03244) find
  34. Zun Wang, Jialu Li, Yicong Hong, Songze Li, Kunchang Li, Shoubin Yu, Yi Wang, Yu Qiao, Yali Wang, Mohit Bansal, and Limin Wang. 2025b. Bootstrapping language-guided navigation learning with self-refining data flywheel. In The Thirteenth International Conference on Learning Representations. find
  35. Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. 2024. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13624–13634. find
  36. Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. 2023. Scaling data generation in vision-and-language navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12009– 12020. find
  37. Tong Xiao and Jingbo Zhu. 2025. Foundations of large language models. arXiv preprint arXiv:2501.09223. — [arXiv:2501.09223](https://arxiv.org/abs/2501.09223) find
  38. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024a. Vision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644. find
  39. Siqi Zhang, Yanyuan Qiao, Qunbo Wang, Longteng Guo, Zhihua Wei, and Jing Liu. 2025a. Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks. arXiv preprint arXiv:2503.13966. — [arXiv:2503.13966](https://arxiv.org/abs/2503.13966) find
  40. Yue Zhang, Quan Guo, and Parisa Kordjamshidi. 2024b. Navhint: Vision and language navigation agent with a hint generator. In Findings of the Association for Computational Linguistics: EACL 2024, pages 92– 103, St. Julian’s, Malta. Association for Computational Linguistics. find
  41. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, and 1 others. 2024a. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1–65. find
  42. Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. 2024b. NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models. arXiv preprint. ArXiv:2407.12366 [cs]. — [arXiv:2407.12366](https://arxiv.org/abs/2407.12366) find
  43. Gengze Zhou, Yicong Hong, Zun Wang, Chongyang Zhao, Mohit Bansal, and Qi Wu. 2024c. Same: Learning generic language-guided visual navigation with state-adaptive mixture of experts. arXiv preprint arXiv:2412.05552. find
  44. Gengze Zhou, Yicong Hong, and Qi Wu. 2023. NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models. arXiv preprint. ArXiv:2305.16986 [cs]. find
  45. Yue Zhang and Parisa Kordjamshidi. 2023. Vln-trans: Translator for the vision and language navigation agent. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13219–13233, Toronto, Canada. Association for Computational Linguisti… find
  46. Yue Zhang, Tianyi Ma, Zun Wang, Yanyuan Qiao, and Parisa Kordjamshidi. 2025b. Vision-and-language navigation with analogical textual descriptions in LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15017–15025, Suzhou, China. Association for Comp… find
  47. Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. 2024c. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. Transactions on Machine Learning Research. Survey Certification. find
  48. Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. 2025c. SPARTUN3d: Situated spatial understanding of 3d world in large language model. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-breaking-down-building-up-mixture/