Elamur Research⌕
☰
home / papers / 2026 / DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents

DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents

Paper · 2026 · Yawen Wang, Yihan Dai, Jianming Chen et al.

Summary

LLM planners in Vision-and-Language Navigation (VLN) are hard to diagnose. Existing explanation tools either find important timesteps or highlight important visual features, but not both together. They also handle the run-to-run randomness of black-box LLM agents poorly, and textual Chain-of-Thought is often unfaithful. The authors propose DEFT, a single interpreter that answers two questions: when a decision mattered and what visual evidence drove it.

DEFT uses a VLN⟳BERT backbone with two heads that share one latent representation. A Mask Head finds critical steps by masking actions counterfactually and measuring how much the reward drops. An Action Head acts as a differentiable stand-in for the agent. Several such surrogates are trained on separate runs of the target agent using an imitation-plus-RL objective, and their Integrated Gradients maps are averaged into stable pixel-level saliency.

On MatterPort3D with NavGPT, MapGPT and NavGPT-2, DEFT outperforms baselines on both temporal and feature fidelity, and a user study supports its practical value. This gives developers a tool to locate the step and the visual cue behind a navigation failure.

Key points

  • On timestep criticality, DEFT reaches RRD of 1.28–4.13 and Fidelity of 0.15–0.77 across the three agents, beating baselines in every case.
  • For feature attribution, it outperforms both white-box and black-box methods on Insertion and Deletion, and its µF scores stay positive.
  • In the user study, participants preferred DEFT's explanations 78% of the time. Their success at diagnosing failure causes rose to 64%, compared with 30% for composite baselines.
  • Surrogate fidelity was measured in nDTW (target self-consistency vs. surrogate agreement):
    • NavGPT: 0.55 vs. 0.60
    • MapGPT: 0.61 vs. 0.58 (the surrogate exceeds the agent's own consistency)
    • NavGPT-2: 0.80 vs. 0.99
  • Code is released at github.com/yihandai/DEFT. Stated limitations are reliance on reward signals and evaluation only on discrete MatterPort3D; continuous settings would need a backbone such as ETP-Nav.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 130 works

Cite

Cite as Yawen Wang, Yihan Dai, Jianming Chen, Junjie Wang and Qing Wang. 2026. DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1363

@inproceedings{wang2026deft,
  title = {{DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents}},
  author = {Yawen Wang, Yihan Dai, Jianming Chen, Junjie Wang and Qing Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1363},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1363}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. 2025. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. IEEE Trans. Pattern Anal. Mach. Intell., 47(7):5130–5145. find
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674–3683. Computer Vision Foundatio…
  3. Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Jun Hu, Qing Wang, and Fanjiang Xu. 2025. Understanding individual agent importance in multi-agent system via counterfactual reasoning. In AAAI, pages 15785–15794. AAAI Press. find
  4. Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee Kenneth Wong. 2024a. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. In ACL (1), pages 9796–9810. Association for Computational Linguistics. find
  5. Ruoyu Chen, Hua Zhang, Siyuan Liang, Jingzhi Li, and Xiaochun Cao. 2024b. Less is more: Fewer interpretable region via submodular subset selection. In ICLR. OpenReview.net. find
  6. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In CVPR, pages 16516–16526. IEEE. find
  7. Zelei Cheng, Xian Wu, Jiahao Yu, Wenhai Sun, Wenbo Guo, and Xinyu Xing. 2023. Statemask: Explaining deep reinforcement learning through state mask. In NeurIPS. find
  8. Zelei Cheng, Xian Wu, Jiahao Yu, Sabrina Yang, Gang Wang, and Xinyu Xing. 2024. RICE: breaking through the training bottlenecks of reinforcement learning with explanation. In ICML. Open-Review.net. find
  9. Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. In ViGIL@NeurIPS. find
  10. Vihan Jain, Gabriel Magalhães, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL (1), pages 1862–1872. Association for Computational Linguistics. find
  11. Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. 2021. Guided integrated gradients: An adaptive path method for removing noise. In CVPR, pages 5050–5058. Computer Vision Foundation / IEEE. find
  12. Shicheng Liu and Minghui Zhu. 2025. UTILITY: utilizing explainable reinforcement learning to improve reinforcement learning. In ICLR. OpenReview.net. find
  13. Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. 2024. Discuss before moving: Visual language navigation via multi-expert discussions. In ICRA, pages 17380–17387. IEEE. find
  14. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928– 1937. JMLR.org. find
  15. Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. 2019. Towards interpretable reinforcement learning using attention augmented agents. In NeurIPS, pages 12329–12338. find
  16. Paul Novello, Thomas Fel, and David Vigouroux. 2022. Making sense of dependence: Efficient black-box explanations using dependence measure. In NeurIPS. find
  17. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  18. CoRR, Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding atari agents. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 1787–1796. PMLR. find
  19. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618–626. IEEE Computer Society.
  20. Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. EDGE: explaining deep reinforcement learning policies. In NeurIPS, pages 12222–12236. find
  21. Suraj Srinivas and François Fleuret. 2021. Rethinking the role of gradient-based attribution methods for model interpretability. In ICLR. OpenReview.net. find
  22. Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo, and Stephen Gould. 2021. VLN BERT: A recurrent vision-and-language BERT for navigation. In CVPR, pages 1643–1653. Computer Vision Foundation / IEEE. find
  23. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR. find
  24. Sandy H. Huang, Kush Bhatia, Pieter Abbeel, and Anca D. Dragan. 2018. Establishing appropriate trust via critical states. In IROS, pages 3929–3936. IEEE. find
  25. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In NeurIPS.
  26. Yuchen Wu, Pengcheng Zhang, Meiying Gu, Jin Zheng, and Xiao Bai. 2024. Embodied navigation with multimodal information: A survey from tasks to methodology. Inf. Fusion, 112:102532. find
  27. Jiahao Yu, Wenbo Guo, Qi Qin, Gang Wang, Ting Wang, and Xinyu Xing. 2023. AIRS: explanation for deep reinforcement learning based security applications. In USENIX Security Symposium, pages 7375–7392. USENIX Association. find
  28. Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding dqns. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 1899–1908. JMLR.org. find
  29. Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. 2024. Navid: Video-based VLM plans the next step for vision-and-language navigation. In Robotics: Science and Systems. find
  30. Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. 2024a. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In ECCV (7), volume 15065 of Lecture Notes in Computer Science, pages 260–278. Springer. find
  31. Gengze Zhou, Yicong Hong, and Qi Wu. 2024b. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In AAAI, pages 7641–7649. AAAI Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-deft-demystifying-vln-failures-unified/