Elamur Research⌕
☰
home / papers / 2026 / VideoPro: Adaptive Program Reasoning for Long Video Understanding

VideoPro: Adaptive Program Reasoning for Long Video Understanding

Paper · 2026 · Chenglin Li, Feng Han, Yikun Wang et al.

Summary

Long videos are hard for VideoLLMs because the visual cues needed to answer a question are few and scattered across a long timeline. Earlier visual-programming approaches, where an LLM writes code that calls perception tools, have three weaknesses: they rely on per-frame modules, run programs for every query, and execute each program once with no way to recover from errors. VideoPro addresses all three. A trained VideoLLM first plans its approach. Easy queries get a single native model call. Harder ones are broken into executable programs over a coarse-to-fine library of video modules. If a program crashes or the answer's confidence is low, the model rewrites and reruns it. Confidence is computed as a length-normalized token probability. Training data comes from programs synthesized by a teacher LLM, sorted by execution outcome. The model is trained with supervised tuning followed by GRPO. VideoPro raises Qwen3-VL-8B by 6.7% on average across long-video benchmarks, with the largest gains on longer clips. This shows that selective tool use can be both accurate and cost-efficient.

Key points

  • Motivation: on LongVideoBench, native VideoLLM answers with confidence above 0.9 are correct over 90% of the time. This suggests that calling programs for every query wastes compute.
  • Module library: alongside basic Python utilities, the library provides five main modules, which handle retrieval across modalities, localization in time, extraction of fine-grained visual details, summarization of global context, and reasoning to generate answers. The native path uses a single query_native call.
  • Training data: programs are labeled as execution failures, successful runs with wrong answers, or correct predictions. These labels form a reason-and-refine supervision set, used for SFT and then GRPO.
  • Main results: 49.7% accuracy on LVBench, surpassing GPT-4o. On Medium (2–15 min) and Long (>15 min) videos, Qwen3-VL-8B gains 5.2 and 5.3 points, while Short-video accuracy stays roughly unchanged.
  • Ablation and limits: refinement alone lifts the SFT-only Qwen3-VL score from 61.2% to 65.0%. The authors note two limitations: the module library is hand-curated, and confidently wrong outputs can bypass refinement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Wuhan UniversityWuhan · China35 works · 1,127 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Chenglin Li, Feng Han, Yikun Wang, Ruilin Li, Shuai Dong, Haowen Hou, Haitao Li, Qianglong Chen, Feng Tao, Jingqi Tong, Yin Zhang and Jiaqi Wang. 2026. VideoPro: Adaptive Program Reasoning for Long Video Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.341

@inproceedings{li2026videopro,
  title = {{VideoPro: Adaptive Program Reasoning for Long Video Understanding}},
  author = {Chenglin Li, Feng Han, Yikun Wang, Ruilin Li, Shuai Dong, Haowen Hou, Haitao Li, Qianglong Chen, Feng Tao, Jingqi Tong, Yin Zhang and Jiaqi Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.341},
  url = {https://doi.org/10.18653/v1/2026.acl-long.341}
}

References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Jinhe Bi, Yifan Wang, Danqi Yan, Aniri, Wenke Huang, Zengjie Jin, Xiaowen Ma, Artur Hecker, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. 2025a. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. Preprint, arXiv:2502.12119. — [arXiv:2502.12119](https://arxiv.org/abs/2502.12119) find
  3. Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025b. LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti…
  4. Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. 2024a. Cg-bench: Clue-grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412.12075. — [arXiv:2412.12075](https://arxiv.org/abs/2412.12075) find
  5. Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, and 1 others. 2024b. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188. find
  6. Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. 2023. Tracking anything with decoupled video segmentation. In ICCV. find
  7. Rohan Choudhury, Koichiro Niinuma, Kris M Kitani, and László A Jeni. 2023. Zero-shot video question answering with procedural programs. arXiv preprint arXiv:2312.00937. — [arXiv:2312.00937](https://arxiv.org/abs/2312.00937) find
  8. Jingfeng Zhang Di Zhao, Hongsheng Hu, Philippe Fournier-Viger, Gillian Dobbie, and Yun Sing Koh. 2025. Balancing invariant and specific knowledge for domain generalization with online knowledge distillation. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligenc… find
  9. Jonathan St BT Evans. 2003. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454–459. find
  10. Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. 2024. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75–92. Springer. find
  11. Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. 2025. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776. — [arXiv:2503.21776](https://arxiv.org/abs/2503.21776) find
  12. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075.
  13. Minghe Gao, Juncheng Li, Hao Fei, Liang Pang, Wei Ji, Guoming Wang, Zheqi Lv, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. 2024. De-fine: De composing and re fin ing visual programs with auto-feedback. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7649–7657. find
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  15. Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14953–14962. find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  17. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  18. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355.
  19. Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122.
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  21. Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. 2025. Videomind: A chain-of-lora agent for long video reasoning. arXiv preprint arXiv:2503.13444. — [arXiv:2503.13444](https://arxiv.org/abs/2503.13444) find
  22. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  23. Ahmad Mahmood, Ashmal Vayani, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan. 2024. Vurf: A general-purpose reasoning and self-refinement framework for video understanding. arXiv preprint arXiv:2403.14743. — [arXiv:2403.14743](https://arxiv.org/abs/2403.14743) find
  24. OpenAI. 2025. Gpt-5. https://openai.com/gpt-5/. Accessed: 2025-12-25. ↗
  25. OpenAI. 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-05-15. ↗
  26. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  27. Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. 2025. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169. find
  28. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, and 1 others. 2023. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449. find
  29. Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. 2024. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.08862. — [arXiv:2408.08862](https://arxiv.org/abs/2408.08862) find
  30. Yiliu Sun, Yanfang Zhang, Zicheng Zhao, Sheng Wan, Dacheng Tao, and Chen Gong. 2025. Fast-slow-thinking: Complex task solving with large language models. arXiv preprint arXiv:2504.08690. — [arXiv:2504.08690](https://arxiv.org/abs/2504.08690) find
  31. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11888– 11898.
  32. ByteDance Seed Team. 2025. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062.
  33. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  34. Dawei Wang, Chengming Zhou, Di Zhao, Xinyuan Liu, Marci Chi Ma, Gary Ushaw, and Richard Davison. 2026a. Towermind: A tower defence game learning environment and benchmark for llm as agents. arXiv preprint arXiv:2601.05899. — [arXiv:2601.05899](https://arxiv.org/abs/2601.05899) find
  35. Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, and 1 others. 2026b. Deepgen 1.0: A lightweight unified multimodal model for advancing image generation and editing. arXiv preprint arXiv:2602.12205. — [arXiv:2602.12205](https://arxiv.org/abs/2602.12205) find
  36. Dianyi Wang, Chaofan Ma, Feng Han, Size Wu, Wei Song, Yibin Wang, Zhixiong Zhang, Tianhang Wang, Siyuan Wang, Zhongyu Wei, and 1 others. 2026c. Unireason 1.0: A unified reasoning framework for world knowledge aligned image generation and editing. arXiv preprint arXiv:2602.02437. — [arXiv:2602.02437](https://arxiv.org/abs/2602.02437) find
  37. Dianyi Wang, Wei Song, Yikun Wang, Siyuan Wang, Kaicheng Yu, Zhongyu Wei, and Jiaqi Wang. 2025a. Autoregressive semantic visual reconstruction helps vlms understand better. arXiv preprint arXiv:2506.09040.
  38. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, and 1 others. 2024a. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035. find
  39. Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2024b. Videoagent: Long-form video understanding with large language model as agent. find
  40. In European Conference on Computer Vision, pages 58–76. Springer. find
  41. Chen. 2024b. Swift:a scalable lightweight infrastructure for fine-tuning. Preprint, arXiv:2408.05517. — [arXiv:2408.05517](https://arxiv.org/abs/2408.05517) find
  42. Yikun Wang, Zuyan Liu, Ziyi Wang, Han Hu, Pengfei Liu, and Yongming Rao. 2025b. Geovista: Web-augmented agentic visual reasoning for geolocalization. Preprint, arXiv:2511.15705. — [arXiv:2511.15705](https://arxiv.org/abs/2511.15705) find
  43. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, and 1 others. 2025a. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13691–13701. find
  44. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2025c. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3272–3283. find
  45. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857. find
  46. Wenyi Xiao, Leilei Gan, Weilong Dai, Wanggui He, Ziwei Huang, Haoyuan Li, Fangxun Shu, Zhelun Yu, Peng Zhang, Hao Jiang, and 1 others. 2025. Fast-slow thinking for large vision-language model reasoning. arXiv preprint arXiv:2504.18458. — [arXiv:2504.18458](https://arxiv.org/abs/2504.18458) find
  47. Wenrui Zhou, Mohamed Hendy, Shu Yang, Qingsong Yang, Zikun Guo, Yuyu Luo, Lijie Hu, and Di Wang. 2025b. Flattery in motion: Benchmarking and analyzing sycophancy in video-llms. ArXiv:2506.07180 [cs.CL].
  48. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2023. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. Preprint, arXiv:2310.01852… — [arXiv:2310.01852](https://arxiv.org/abs/2310.01852) find
  49. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  50. Junqi Yang, Yuecong Min, Jie Zhang, Shiguang Shan, and Xilin Chen. 2026. Infact: A diagnostic benchmark for induced faithfulness and factuality hallucinations in video-llms. arXiv preprint arXiv:2603.11481.
  51. Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. 2023a. A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235. find
  52. Hang Zhang, Xin Li, and Lidong Bing. 2023b. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858.
  53. Shengjia Zhang, Junjie Wu, Jiawei Chen, Changwang Zhang, Xingyu Lou, Wangchunshu Zhou, Sheng Zhou, Can Wang, and Jun Wang. 2025. Othink-r1: Intrinsic fast/slow thinking mode switching for over-reasoning mitigation. arXiv preprint arXiv:2506.02397. — [arXiv:2506.02397](https://arxiv.org/abs/2506.02397) find
  54. Di Zhao, Yun Sing Koh, Gillian Dobbie, Hongsheng Hu, and Philippe Fournier-Viger. 2024a. Symmetric self-paced learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16961–16969. find
  55. Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2026-videopro-adaptive-program-reasoning-long/