Elamur Research⌕
☰
home / papers / 2026 / VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions

VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions

Paper · 2026 · Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh et al.

Summary

Standard Vision-and-Language Navigation benchmarks assume every instruction is feasible, so agents are never asked to admit that a requested object is missing. This paper addresses false-premise instructions, where the named target does not exist in the indicated room. The authors build VLN-NF on top of REVERIE in Matterport3D. An LLM swaps the original target for a plausible but absent object, and a VLM confirms the object is not there. Agents must reach the room, search it, and then declare NOT-FOUND when the target is absent. Ordinary SPL rewards stopping early, so the authors introduce REV-SPL. It scores room reaching, exploration coverage against a reference search protocol, and decision correctness across mixed feasible and infeasible episodes. They also propose ROAM, a two-stage hybrid. A weakly supervised model first finds the target room. LLM/VLM-driven exploration, guided by a free-space clearance prior, then searches the room and verifies the target. ROAM scores best among the compared methods, while baselines tend to under-explore and give up too soon. This makes the work a step toward robots that handle mistaken human requests safely.

Key points

  • Dataset size: VLN-NF pairs each original REVERIE instruction (FOUND) with a generated infeasible counterpart (NF), sharing the same start point and target room.
    • Train: 55 scans, 2,362 pairs, 4,724 instructions.
    • Val-seen: 38 scans, 234 pairs.
    • Val-unseen: 10 scans, 718 pairs, 1,436 instructions.
  • Label quality: Human checks of a sample found NOT-FOUND labels about 98% correct (under 2% errors). Strict filtering keeps only 45% of REVERIE episodes.
  • Main results (REV-SPL): ROAM reaches 6.1. The supervised DUET-style baseline (Chen et al., 2022) gets 4.2, and two LLM-based agents get 1.0 and 1.5. ROAM's gain over the supervised baseline is about 45%, and it needs only room-level annotations.
  • Failure mode identified: Simply adding a NOT-FOUND action and training by imitation leads to premature abstention, because there is no clear signal for when a search is finished.
  • Limitations:
    • ROAM depends on a fixed Grounding-DINO confidence threshold.
    • Only the "object absent from the specified room" type of false premise is covered.
    • The system has no recovery behavior, such as asking the user for clarification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan UniversityTaipei · Taiwan29 works · 2,116 citations🎓National Tsing Hua UniversityHsinchu · Taiwan8 works · 834 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksAutomotive Engineering · 6 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh, Min Sun and Winston H. Hsu. 2026. VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1152

@inproceedings{su2026vln,
  title = {{VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions}},
  author = {Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh, Min Sun and Winston H. Hsu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1152},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1152}
}

References

40 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537–16547. find
  2. Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. 2024. Spatialrgpt: Grounded spatial reasoning in vision-language models. In Advances in Neur… find
  3. Iulia Comsa and Srini Narayanan. 2023. A benchmark for reasoning with spatial prepositions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16328–16335, Singapore. Association for Computational Linguistics. find
  4. Georgios Georgakis, Karl Schmeckpeper, Karan Wan-choo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. 2022. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15460–15470. find
  5. Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  6. Meera Hahn, Amit Raj, and James M. Rehg. 2023. Which way is ‘right’?: Uncovering limitations of vision-and-language navigation models. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 2415–2417, Richland, SC. find
  7. Haodong Hong, Sen Wang, Zi Huang, Qi Wu, and Jiajun Liu. 2024. Navigating beyond instructions: Vision-and-language navigation in obstructed environments. arXiv preprint arXiv:2407.21452. Accepted to ACM Multimedia (MM) 2024. — [arXiv:2407.21452](https://arxiv.org/abs/2407.21452) find
  8. Wen-Han Hsieh, Elvis Hsieh, Dantong Niu, Trevor Darrell, Roei Herzig, and David M. Chan. 2025. Do what? teaching vision-language-action models to reject the impossible. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11861–11869, Suzhou, China. Association for Computa… find
  9. Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. 2023. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. In Proceedings of the IEEE/CVF Conference on Computer Vis… find
  10. Po-Chen Ko, Hung-Ting Su, CY Chen, Jia-Fong Yeh, Min Sun, and Winston H. Hsu. 2024. Context-aware replanning with pre-explored semantic map for object navigation. In 8th Annual Conference on Robot Learning. find
  11. Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, and Si Liu. 2024. Controllable navigation instruction generation with chain of thought prompting. In European Conference on Computer Vision, pages 37–54. Springer. find
  12. Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV). find
  13. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954. find
  14. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, and 1 others. 2022. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965– 109… find
  15. Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. 2025. Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  16. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, and 1 others. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer.
  17. Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. 2024. Discuss before moving: Visual language navigation via multi-expert discussions. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE. find
  18. OpenAI. 2023. Gpt-4 technical report. Accessed: 2025-05-18.
  19. Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. 2023. Langnav: Language as a perceptual representation for navigation. arXiv preprint arXiv:2310.07889. find
  20. Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942– 15952. find
  21. Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982–9991. find
  22. Yanyuan Qiao, Qianyi Liu, Jiajun Liu, Jing Liu, and Qi Wu. 2024. Llm as copilot for coarse-grained vision-and-language navigation. In European Conference on Computer Vision, pages 459–476. Springer. find
  23. Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, and 1 others. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159. — [arXiv:2401.14159](https://arxiv.org/abs/2401.14159) find
  24. Seokin Seo, Byung-Jun Lee, Jongmin Lee, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. 2024. Mitigating covariate shift in behavioral cloning via robust stationary distribution correction. In Advances in Neural Information Processing Systems, volume 37. find
  25. Francesco Taioli, Stefano Rosa, Alberto Castellini, Lorenzo Natale, Alessio Del Bue, Alessandro Farinelli, Marco Cristani, and Yiming Wang. 2024. Mind the error! detection and localization of instruction errors in vision-and-language navigation. In 2024 IEEE/RSJ International Conference on Intellige… find
  26. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394–406. PMLR. find
  27. Hongbin Wang, Todd R Johnson, Jiajie Zhang, and Yue Wang. 2002. A study of object-location memory. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 24. find
  28. Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. 2023a. Lana: A language-capable navigator for instruction following and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19048–19058. find
  29. Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. 2023b. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636. find
  30. Tao Wu, Chuhao Zhou, Yen Heng Wong, Lin Gu, and Jianfei Yang. 2024. Noisyeqa: Benchmarking embodied question answering against noisy queries. arXiv preprint arXiv:2412.10726. — [arXiv:2412.10726](https://arxiv.org/abs/2412.10726) find
  31. Tao Wu, Chuhao Zhou, Guangyu Zhao, Haozhi Cao, Yewen Pu, and Jianfei Yang. 2025. When robots should say “i don’t know”: Benchmarking abstention in embodied question answering. arXiv preprint arXiv:2512.04597. — [arXiv:2512.04597](https://arxiv.org/abs/2512.04597) find
  32. Jiannan Xiang, Xin Wang, and William Yang Wang. 2020. Learning to stop: A simple yet effective approach to urban vision-language navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 699–707, Online. Association for Computational Linguistics. find
  33. He Yan, Xinyao Hu, Xiangpeng Wan, Chengyu Huang, Kai Zou, and Shiqi Xu. 2023. Inherent limitations of gpt=4 regarding spatial information. Preprint, arXiv:2312.03042. — [arXiv:2312.03042](https://arxiv.org/abs/2312.03042) find
  34. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. find
  35. Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2024. Thinking in space: How multimodal large language models see, remember, and recall spaces. Preprint, arXiv:2412.14171. find
  36. Xiaoyu Zhang, Matthew Chang, Pranav Kumar, and Saurabh Gupta. 2024a. Diffusion meets DAgger: Supercharging eye-in-hand imitation learning. In Robotics: Science and Systems (RSS). find
  37. Yue Zhang, Quan Guo, and Parisa Kordjamshidi. 2024b. Navhint: Vision and language navigation agent with a hint generator. Association for Computational Linguistics. find
  38. Yue Zhang and Parisa Kordjamshidi. 2023. Vln-trans: Translator for the vision and language navigation agent. arXiv preprint arXiv:2302.09230. find
  39. Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. 2024a. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In European Conference on Computer Vision, pages 260–278. Springer. find
  40. Gengze Zhou, Yicong Hong, and Qi Wu. 2024b. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2026-vln-nf-feasibility-aware-vision/