Elamur Research⌕
☰
home / papers / 2026 / ProgressLM: Towards Progress Reasoning in Vision-Language Models

ProgressLM: Towards Progress Reasoning in Vision-Language Models

Paper · 2026 · Jianshu Zhang, Chengxuan Qian, Haosen Sun et al.

Summary

The paper asks whether vision-language models can judge how far a task has advanced when they see only one intermediate frame plus a demonstration of the full task. The authors build Progress-Bench on RoboMind robotic manipulation data. In each instance, a model compares a single observation against a demonstration and outputs a progress percentage, or N/A when the two do not match. The benchmark varies three factors: demonstration format (key frames or step-by-step text), camera viewpoint (same or different) and answerability.

Borrowing from human cognition, the authors split the task into two stages. The model first retrieves the demonstration step that best matches the observation, then reasons about how far the scene has moved past or before that anchor. They test this both as a structured prompt and as a training recipe using the automatically built ProgressLM-45K dataset (supervised cold-start followed by reinforcement learning).

Across 14 VLMs, progress estimation proves unreliable, and prompting helps only some models. In contrast, the small trained ProgressLM-3B improves consistently and matches or beats GPT-5. Possible uses include reward signals for reinforcement learning and anomaly detection.

Key points

  • Progress-Bench contains 3,325 sampled observations from 240 trajectories. Ground-truth progress is linearly interpolated between annotated key steps. A manual check of 100 trajectories found 86% fully smooth.
  • Unanswerable cases are created by editing either the demonstration or the observation (e.g., swapping the object), so that the correct answer is N/A.
  • ProgressLM-45K comprises about 25K chain-of-thought samples for SFT and about 20K for RL. It covers several robot embodiments: Franka, UR5e, AgileX Dual-Arm and TienKung Humanoid.
  • Model outputs follow a tagged schema: <ref_think>/<ref> for the retrieval step and <score_think>/<score> for the simulation step.
  • Text-based demonstrations are harder because the model must accumulate hidden state changes across steps. A 7B variant also gains from the training even after only one RL epoch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Santa BarbaraSanta Barbara · United States57 works · 9,752 citations🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Jianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu, Dingcheng Wang, Letian Xue and Han Liu. 2026. ProgressLM: Towards Progress Reasoning in Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.516

@inproceedings{zhang2026progresslm,
  title = {{ProgressLM: Towards Progress Reasoning in Vision-Language Models}},
  author = {Jianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu, Dingcheng Wang, Letian Xue and Han Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.516},
  url = {https://doi.org/10.18653/v1/2026.acl-long.516}
}

References

62 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minttu Alakuijala, Reginald McLean, Isaac Woungang, Nariman Farsad, Samuel Kaski, Pekka Marttinen, and Kai Yuan. 2025. Video-language critic: Transferable reward functions for language-conditioned robotics. Transactions on Machine Learning Research (TMLR). find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  3. Qianzhong Chen, Justin Yu, Mac Schwager, Pieter Abbeel, Fred Shentu, and Philipp Wu. 2025. Sarm: Stage-aware reward modeling for long horizon robot manipulation. Preprint, arXiv:2509.25358. — [arXiv:2509.25358](https://arxiv.org/abs/2509.25358) find
  4. Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. 2022. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for dynamics and control conference, pages 893–905. PMLR. find
  5. Yilong Dai, Ziyi Wang, Chenguang Wang, Kexin Zhou, Yiheng Qian, Susu Xu, and Xiang Yan. 2026. Persona-aware and explainable bikeability assessment: A vision-language model approach. Preprint, arXiv:2601.03534. — [arXiv:2601.03534](https://arxiv.org/abs/2601.03534) find
  6. Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. 2025. Racer: Rich language-guided failure recovery policies for imitation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 15657–15664. IEEE. find
  7. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  8. Yixuan Du, Chenxiao Yu, Haoyan Xu, Ziyi Wang, Yue Zhao, and Xiyang Hu. 2026. Multimodal generative engine optimization: Rank manipulation for vision-language model rankers. Preprint, arXiv:2601.12263. find
  9. Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. 2024. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation. arXiv preprint arXiv:2410.00371. find
  10. Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. 2023. Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems, 36:68760–68783. find
  11. Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343– 18362. find
  12. Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. 2025. Webevolver: Enhancing web agent self-improvement with co-evolving world model. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8970– 8986.
  13. Haonan Ge, Yiwei Wang, Kai-Wei Chang, Hang Wu, and Yujun Cai. 2025a. Framemind: Frame-interleaved video reasoning via reinforcement learning. arXiv preprint arXiv:2509.24008. — [arXiv:2509.24008](https://arxiv.org/abs/2509.24008) find
  14. Haonan Ge, Yiwei Wang, Ming-Hsuan Yang, and Yujun Cai. 2025b. Mrfd: Multi-region fusion decoding with self-consistency for mitigating hallucinations in lvlms. arXiv preprint arXiv:2508.10264. find
  15. Kai Han, Chongwen Lyu, Lele Ma, Chengxuan Qian, Siqi Ma, Zheng Pang, Jun Chen, and Zhe Liu. 2025. Climd: A curriculum learning framework for imbalanced multimodal diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 65–74. Springer. find
  16. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. 2025. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computatio…
  17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  18. Minda Hu, Tianqing Fang, Jianshu Zhang, Junyu Ma, Zhisong Zhang, Jingyan Zhou, Hongming Zhang, Haitao Mi, Dong Yu, and Irwin King. 2025. Webcot: Enhancing web agent reasoning by reconstructing chain-of-thought in reflection, branching, and rollback. arXiv preprint arXiv:2505.20013, 7. find
  19. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  20. Mengzhao Jia, Wenhao Yu, Kaixin Ma, Tianqing Fang, Zhihan Zhang, Siru Ouyang, Hongming Zhang, Dong Yu, and Meng Jiang. 2024. Leopard: A vision language model for text-rich multi-image tasks. arXiv preprint arXiv:2410.01744. — [arXiv:2410.01744](https://arxiv.org/abs/2410.01744) find
  21. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  22. Young-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang, Byungsoo Ko, Han-Gyu Kim, Dongyu Yao, Xuankun Rong, Eojin Joo, Seung-Ho Han, and 1 others. 2025. Multiverse: A multi-turn conversation benchmark for evaluating large vision and language models. In Proceedings of the IEEE/CVF International C… find
  23. Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J Lim. 2021. Generalizable imitation learning from observation via inferring goal proximity. Advances in neural information processing systems, 34:16118–16130. find
  24. Mukai Li, Qingcheng Zeng, Tianqing Fang, Zhenwen Liang, Linfeng Song, Qi Liu, Haitao Mi, and Dong Yu. 2026. Verified critical step optimization for llm agents. arXiv preprint arXiv:2602.03412. find
  25. Weifeng Lu, Minghao Ye, Zewei Ye, Ruihan Tao, Shuo Yang, and Bo Zhao. 2025. Robofac: A comprehensive framework for robotic failure analysis and correction. arXiv preprint arXiv:2505.12224. — [arXiv:2505.12224](https://arxiv.org/abs/2505.12224) find
  26. Zhen Luo, Yixuan Yang, Yanfu Zhang, and Feng Zheng. 2025. Roboreflect: A robotic reflective reasoning framework for grasping ambiguous-condition objects. arXiv preprint arXiv:2501.09307. — [arXiv:2501.09307](https://arxiv.org/abs/2501.09307) find
  27. Jason Ma, William Liang, Hung-Ju Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. 2024a. Dreureka: Language model guided sim-toreal transfer. RSS. find
  28. Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, and 1 others. 2024b. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations. find
  29. Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. 2023. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pages 23301–23320. PMLR. find
  30. Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. 2022. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030. — [arXiv:2210.00030](https://arxiv.org/abs/2210.00030) find
  31. Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, and 1 others. 2018. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX symposium on operating systems design and implement… find
  32. Yi Nian, Shenzhe Zhu, Yuehan Qin, Li Li, Ziyi Wang, Chaowei Xiao, and Yue Zhao. 2025. Jaildam: Jailbreak detection with adaptive memory for vision-language model. Preprint, arXiv:2504.03770. — [arXiv:2504.03770](https://arxiv.org/abs/2504.03770) find
  33. OpenAI. 2024. Gpt-5 system card. https://openai.com/index/gpt-5-system-card/. ↗
  34. Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. 2024. Image textualization: An automatic framework for creating accurate and detailed image descriptions. arXiv preprint arXiv:2406.07502. — [arXiv:2406.07502](https://arxiv.org/abs/2406.07502) find
  35. Chengxuan Qian, Kai Han, Jiaxin Liu, Zhenlong Yuan, Zhengzhong Zhu, Jingchao Wang, Chongwen Lyu, Jun Chen, and Zhe Liu. 2025a. Dyncim: Dynamic curriculum for imbalanced multimodal learning. arXiv preprint arXiv:2503.06456. — [arXiv:2503.06456](https://arxiv.org/abs/2503.06456) find
  36. Chengxuan Qian, Shuo Xing, Shawn Li, Yue Zhao, and Zhengzhong Tu. 2025b. Decalign: Hierarchical cross-modal alignment for decoupled multimodal representation learning. arXiv preprint arXiv:2503.11892. — [arXiv:2503.11892](https://arxiv.org/abs/2503.11892) find
  37. Daniel L Schacter, Donna Rose Addis, and Randy L Buckner. 2008. Episodic simulation of future events: Concepts, data, and applications. Annals of the New York Academy of Sciences, 1124(1):39–60. find
  38. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  39. Charles Spearman. 1961. The proof and measurement of association between two things. find
  40. Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Yixuan Li, Jason E Weston, and Ping Yu. 2025. Hybrid reinforcement: When reward is sparse, it’s better to be dense. arXiv preprint arXiv:2510.07242. — [arXiv:2510.07242](https://arxiv.org/abs/2510.07242) find
  41. Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, and 1 others. 2025a. Vagen: Reinforcing world model reasoning for multi-turn vlm agents. arXiv preprint arXiv:2510.16907. find
  42. Ziyi Wang, Qizan Guo, Rishitosh Singh, and Xiyang Hu. 2026b. Do vision language models understand human engagement in games? Preprint, arXiv:2603.18480. — [arXiv:2603.18480](https://arxiv.org/abs/2603.18480) find
  43. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, and 20 others. 2025a. Qwen-image technical report. Preprint, arXiv:25…
  44. Hang Wu, Yujun Cai, Haonan Ge, Hongkai Chen, Ming-Hsuan Yang, and Yiwei Wang. 2025b. Re-fineshot: Rethinking cinematography understanding with foundational skill evaluation. arXiv preprint arXiv:2510.02423. — [arXiv:2510.02423](https://arxiv.org/abs/2510.02423) find
  45. Hang Wu, Yujun Cai, Zehao Li, Haonan Ge, Bowen Sun, Junsong Yuan, and Yiwei Wang. 2026. Cam-reasoner: Reinforcing camera movement understanding via structured spatial reasoning. arXiv preprint arXiv:2602.00181. — [arXiv:2602.00181](https://arxiv.org/abs/2602.00181) find
  46. Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, and 1 others. 2025c. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. In Robotics: Science and Systems (RSS) 2025. Robotics: Science and… find
  47. Haotian Xia, Haonan Ge, Junbo Zou, Hyun Woo Choi, Xuebin Zhang, Danny Suradja, Botao Rui, Ethan Tran, Wendy Jin, Zhen Ye, and 1 others. 2025. Sportr: A benchmark for multimodal large language model reasoning in sports. arXiv preprint arXiv:2511.06499. — [arXiv:2511.06499](https://arxiv.org/abs/2511.06499) find
  48. Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, and 1 others. 2025b. Explore to evolve: Scaling evolved aggregation logic via proactive online exploration for deep research agents. arXiv preprint arXiv:2510.14438. — [arXiv:2510.14438](https://arxiv.org/abs/2510.14438) find
  49. Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, and Zhengzhong Tu. 2025. Re-align: Aligning vision language models via retrieval-augmented direct preference optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Langu… find
  50. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025c. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  51. Daniel Yang, Davin Tjia, Jacob Berg, Dima Damen, Pulkit Agrawal, and Abhishek Gupta. 2024. Rank2reward: Learning shaped reward functions from passive video. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2806–2813. IEEE. find
  52. Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. 2024. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03681. — [arXiv:2402.03681](https://arxiv.org/abs/2402.03681) find
  53. Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, and 1 others. 2025. Spatial mental modeling from limited views. In Structural Priors for Vision Workshop at ICCV’25. find
  54. Ziyi Wang, Yilong Dai, Duanya Lyu, Mateo Nader, Sihan Chen, Wanghao Ye, Zjian Ding, and Xiang Yan. 2026a. Streetdesignai: A multi-persona evaluation system for inclusive infrastructure design. Preprint, arXiv:2601.15671. — [arXiv:2601.15671](https://arxiv.org/abs/2601.15671) find
  55. Zhenlong Yuan, Chengxuan Qian, Jing Tang, Rui Chen, Zijian Song, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li. 2025a. Autodrive-r2: Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving. arXiv preprint arXiv:2509.01944. — [arXiv:2509.01944](https://arxiv.org/abs/2509.01944) find
  56. Zhenlong Yuan, Xiangyan Qu, Chengxuan Qian, Rui Chen, Jing Tang, Lei Sun, Xiangxiang Chu, Dapeng Zhang, Yiwei Wang, Yujun Cai, and 1 others. 2025b. Video-star: Reinforcing open-vocabulary action recognition with tools. arXiv preprint arXiv:2510.08480. — [arXiv:2510.08480](https://arxiv.org/abs/2510.08480) find
  57. Zhenlong Yuan, Xiangyan Qu, Jing Tang, Rui Chen, Lei Sun, Ruidong Chen, Hongwei Yu, Chengxuan Qian, Xiangxiang Chu, Shuo Li, and 1 others. 2026. What if agents could imagine? reinforcing open-vocabulary hoi comprehension through generation. arXiv preprint arXiv:2602.11499. — [arXiv:2602.11499](https://arxiv.org/abs/2602.11499) find
  58. Shaopeng Zhai, Qi Zhang, Tianyi Zhang, Fuxian Huang, Haoran Zhang, Ming Zhou, Shengzhe Zhang, Litao Liu, Sixu Lin, and Jiangmiao Pang. 2025. A vision-language-action-critic model for robotic real-world reinforcement learning. arXiv preprint arXiv:2509.15937. — [arXiv:2509.15937](https://arxiv.org/abs/2509.15937) find
  59. Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, and Yi R Fung. 2025. Vlm2-bench: A closer look at how well vlms implicitly link explicit matching visual cues. arXiv preprint arXiv:2502.12084.
  60. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, and 1 others. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277. find
  61. Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. 2025. Easyr1: An efficient, scalable, multi-modality rl training framework. find
  62. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-progresslm-progress-reasoning-vision-language/