Elamur Research⌕
☰
home / papers / 2026 / RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation

RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation

Paper · 2026 · Chenduo Ying, Linkang Du, Yuanchao Shu et al.

Summary

VLM-based failure detectors in robotic manipulation usually judge a task only after it ends. By then a failure may already have damaged hardware or endangered people. These detectors also often confuse visually similar error types. RoboFailRing addresses both problems.

Offline, it builds a failure memory from simulated failure trajectories generated with FailGen on RLBench. Each stored entry pairs a CLIP embedding of the task's initial frame with one of the final failure frame, and links it to task metadata. During execution, embeddings from multiple camera views are matched against this memory. If a match is similar enough, the robot stops immediately.

A neuro-symbolic grounding step then turns the terse metadata into natural-language context. This context goes into a failure report, together with reference images, that prompts a VLM to explain the cause and suggest a repair.

Across two large simulated datasets, the system detects out-of-distribution failures about 80% of the time and needs roughly half the baseline detection time. On a real robot, it improves VLM reasoning accuracy by about 35%. This moves failure handling from post-hoc analysis toward proactive warning.

Key points

  • Evaluation covers over 6,000 failure trajectories spanning 81 manipulation tasks. FailGen produces 7 failure categories, including No_Grasp, Slip and Translation errors.
  • In real-world tests, Gemini-2.5-flash ran 100 reasoning attempts per failure mode, with and without RoboFailRing; average accuracy rose by about 35%.
  • Ablations:
    • Swapping grounded language for raw symbolic labels drops reasoning accuracy from 68.14% to 32.04%.
    • Multi-view fusion reaches 88.63% recall, versus 41.62% for front-view only.
    • ViT-L/14 adds only +1.67% recall over ViT-B/32 but raises latency from ~14 ms to ~37 ms.
  • Total detection latency is about 15 ms (~66 Hz): 4 ms preprocessing, 10 ms CLIP encoding and 1 ms retrieval. VLM report generation (~3 s) runs asynchronously, after the robot has stopped.
  • Limitations: the start/end-frame representation misses failures that show up in motion over time rather than in the final scene. Recovery execution is not implemented, and only single-arm setups were tested. Code is available at github.com/DynamicPoet/RoboFailRing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksControl and Systems Engineering · 62 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as Chenduo Ying, Linkang Du, Yuanchao Shu and Peng Cheng. 2026. RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.602

@inproceedings{ying2026robofailring,
  title = {{RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation}},
  author = {Chenduo Ying, Linkang Du, Yuanchao Shu and Peng Cheng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.602},
  url = {https://doi.org/10.18653/v1/2026.acl-long.602}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel L… find
  2. Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. 2025. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684– 1704.
  3. Muhayy Ud Din, Waseem Akram, Lyes Saad Saoud, Jan Rosell, and Irfan Hussain. 2025. Vision language action models in robotic manipulation: A systematic review. Preprint, arXiv:2507.10672. — [arXiv:2507.10672](https://arxiv.org/abs/2507.10672) find
  4. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2026. The faiss library. IEEE Transactions on Big Data, 12(2):346–361.
  5. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zen… find
  6. Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. 2023. Vision-language models as success detectors. In Proceedings of the 2nd Conference on Lifelong Learning Agents, pages 120–136. find
  7. Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. 2025. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation. In The 13th International Conference on Learning Repres… find
  8. Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. 2024. Manipulate-anything: Automating real-world robots using vision-language models. In 8th Annual Conference on Robot Learning, pages 5326–5350. find
  9. Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, and Mac Schwager. 2025. Foundation models in robotics: Applications, challenges, and the future. The Internati… find
  10. Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation, pages 12462–12469. find
  11. Huy Ha, Pete Florence, and Shuran Song. 2023. Scaling up and distilling down: Language-guided robot skill acquisition. In 7th Annual Conference on Robot Learning. find
  12. Siyuan Huang, Haonan Chang, Yuhan Liu, Yimeng Zhu, Hao Dong, Abdeslam Boularias, Peng Gao, and Hongsheng Li. 2024. A3vlm: Actionable articulation-aware vision language model. In 8th Annual Conference on Robot Learning, pages 1675– 1690. find
  13. Muhammad A Muttaqien, Tomohiro Motoda, Ryo Hanai, and Yukiyasu Domae. 2025. Visual prompting for robotic manipulation with annotation-guided pick-and-place using act. In 2025 IEEE 21st International Conference on Automation Science and Engineering, pages 2642–2649. find
  14. Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. 2023. Voxposer: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning. find
  15. Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. 2024. Octo: An open-source… find
  16. Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. 2020. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026. find
  17. Byeonghwi Kim, Jinyeon Kim, Yuyeong Kim, Cheol-hong Min, and Jonghyun Choi. 2023. Context-aware planning and environment-aware memory for instruction following embodied agents. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10936–10946. find
  18. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation, pages 9493–9500. find
  19. Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. 2024. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024. find
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  21. Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. 2024. Gensim: Generating robotic simulation tasks via large language models. In The 12th International Conference on Learning Representations. wpumacay. 2024. maniskill-failgen: A… find
  22. Chenduo Ying, Linkang Du, Peng Cheng, and Yuanchao Shu. 2025. Roboinspector: Unveiling the unreliability of policy code for llm-enabled robotic manipulation. Preprint, arXiv:2508.21378. — [arXiv:2508.21378](https://arxiv.org/abs/2508.21378) find
  23. Zeyi Liu, Arpit Bahety, and Shuran Song. 2023. Reflect: Summarizing robot experiences for failure explanation and correction. In 7th Annual Conference on Robot Learning. find
  24. Minjong Yoo, Jinwoo Jang, Wei-Jin Park, and Honguk Woo. 2024. Exploratory retrieval-augmented planning for continual embodied instruction following. In The 39th Annual Conference on Neural Information Processing Systems, pages 67034–67060. find
  25. Zixian Liu, Mingtong Zhang, and Yunzhu Li. 2025. Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation. In 2025 IEEE International Conference on Robotics and Automation, pages 10561–10569. find
  26. Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. 2024. Octopi: Object property reasoning with large tactile-language models. In Proceedings of Robotics: Science and Systems. find
  27. Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. 2025. Vision language models are in-context value learners. In The 13th International Conf… find
  28. Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. 2023. Vip: Towards universal visual reward and representation via value-implicit pre-training. In The 11th International Conference on Learning Representations. find
  29. Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. 2024. Robopoint: A vision-language model for spatial affordance prediction in robotics. In 8th Conference on Robot Learning, volume 270, pages 4005–4020. find
  30. Yi Zhang, Che Liu, Xiancong Ren, Hanchu Ni, Shuai Zhang, Zeyuan Ding, Jiayu Hu, Hanzhe Shan, Zhenwei Niu, Zhaoyang Liu, Shuang Liu, Yue Zhao, Junbo Qi, Qinfan Zhang, Dengjie Li, Yidong Wang, Jiachen Luo, Yong Dai, Zenglin Xu, Bin Shen, Qifan Wang, Jian Tang, and Xiaozhu Ju. 2025. Pelican-vl 1.0: A f… find
  31. Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2025. A review on edge large language models: Design, execution, and applications. ACM Computing Surveys, 57(8):1–35. find
  32. Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. 2024. Retrieval-augmented embodied agents. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17985–17995. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ying-2026-robofailring-retrieval-augmented-language-grounding/