Elamur Research⌕
☰
home / papers / 2026 / MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding

MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding

Paper · 2026 · Fuwen Luo, Shengfeng Lou, Chi Chen et al.

Summary

Multimodal LLMs remain weak at fine-grained temporal reasoning in video, such as locating when events happen. Earlier RL approaches gave only modest gains. The authors attribute this to two causes: training relies on single-segment grounding, and rewards check final answers but ignore whether the reasoning uses time cues. They propose MUSEG, an RL method built on GRPO with Qwen2.5-VL-7B as the base model. It trains on multi-segment grounding, where one query can map to several video intervals. Its reward has two parts. A segment matching reward combines a global overlap score with a local, per-segment match. A timestamp reward encourages the model's reasoning to cite the predicted times. Training is phased: the timestamp reward guides the model early, then is dropped so the model can explore freely. MUSEG-7B improves on temporal grounding benchmarks and transfers to time-sensitive video QA. This suggests that both task design and reward design matter when applying RL to temporal video understanding.

Key points

  • Motivation: A manual check of 50 single-segment queries from E.T. Bench found 15 (about 30%) solvable by recognizing key objects alone. Only 4 of 50 multi-segment queries allowed such a shortcut.
  • Segment matching reward: It averages a global IoU-style term with local per-segment matching (NGIoU). Sequential local matching worked better than maximum matching. Local matching also keeps the number of predicted segments closer to the ground truth.
  • Phased training: Training runs 900 steps in total. The first 400 use the timestamp reward and the remaining 500 omit it. This schedule beat both never using the timestamp reward and using it throughout, across several total step counts.
  • Final MUSEG-7B scores:
    • Charades-STA 59.7, THUMOS14 29.7, THUMOS15 29.3.
    • E.T. Bench subset average 35.9 (REF 60.8, GND 38.8, CAP 25.1, COM 19.0).
    • Training without the timestamp reward gave 56.9, 28.4, 28.3 and an E.T. Bench average of 32.1.
  • Limitations and resources: Training uses only temporal grounding data. The code is available at github.com/THUNLP-MT/MUSEG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🏢Alibaba GroupHangzhou · China201 works · 17,172 citations

Also on the byline: Zhejiang Sci-Tech University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang and Yang Liu. 2026. MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1644

@inproceedings{luo2026museg,
  title = {{MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding}},
  author = {Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang and Yang Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1644},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1644}
}

References

43 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. 2021. find
  2. ViViT: A Video Vision Transformer. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816–6826. find
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.
  4. Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, and 1 others. 2024. Tempo-ralbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818. — [arXiv:2410.10818](https://arxiv.org/abs/2410.10818) find
  5. Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. 2024. CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding. arXiv preprint arXiv:2412.12075. 35557 9 — [arXiv:2412.12075](https://arxiv.org/abs/2412.12075) find
  6. Xinlong Chen, Yuanxing Zhang, Yushuo Guan, Bohan Zeng, Yang Shi, Sihan Yang, Pengfei Wan, Qiang Liu, Liang Wang, and Tieniu Tan. 2025a. VersaVid-R1: A Versatile Video Understanding and Reasoning Model from Question Answering to Captioning Tasks. arXiv preprint arXiv:2506.09079. — [arXiv:2506.09079](https://arxiv.org/abs/2506.09079) find
  7. Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, and Xihui Liu. 2025b. Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1. arXiv preprint arXiv:2503.24376. — [arXiv:2503.24376](https://arxiv.org/abs/2503.24376) find
  8. Jen-Hao Cheng, Vivian Wang, Huayu Wang, Huapeng Zhou, Yi-Hao Peng, Hou-I Liu, Hsiang-Wei Huang, Kuang-Ming Chen, Cheng-Yen Yang, Wenhao Chai, and 1 others. 2025a. TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action. arXiv preprint arXiv:2505.01583. — [arXiv:2505.01583](https://arxiv.org/abs/2505.01583) find
  9. Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, and Shaogang Gong. 2025b. V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning. arXiv preprint arXiv:2503.11495. — [arXiv:2503.11495](https://arxiv.org/abs/2503.11495) find
  10. Jisheng Dang, Jingze Wu, Teng Wang, Xuanhui Lin, Nannan Zhu, Hongbo Chen, Wei-Shi Zheng, Meng Wang, and Tat-Seng Chua. 2025. Reinforcing Video Reasoning with Focused Thinking. arXiv preprint arXiv:2505.24718. — [arXiv:2505.24718](https://arxiv.org/abs/2505.24718) find
  11. Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. 2025. Video-R1: Reinforcing Video Reasoning in MLLMs. arXiv preprint arXiv:2503.21776. — [arXiv:2503.21776](https://arxiv.org/abs/2503.21776) find
  12. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2025. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In Proceedings of the Computer Vision and Pattern Reco…
  13. Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. TALL: Temporal Activity Localization via Language Query. In Proceedings of the IEEE international conference on computer vision, pages 5267–5275. find
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948.
  15. Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. 2024. Online Video Understanding: A Comprehensive Benchmark and Memory-Augmented Method. arXiv preprint arXiv:2501.00584. — [arXiv:2501.00584](https://arxiv.org/abs/2501.00584) find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276.
  17. H. Idrees, A. R. Zamir, Y. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah. 2017. The THUMOS challenge on action recognition for videos “in the wild”. Computer Vision and Image Understanding, 155:1–23. find
  18. Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. 2025a. LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding. arXiv preprint arXiv:2501.08282. — [arXiv:2501.08282](https://arxiv.org/abs/2501.08282) find
  19. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22195– 22206.
  20. Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. 2025b. VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning. arXiv preprint arXiv:2504.06958. — [arXiv:2504.06958](https://arxiv.org/abs/2504.06958) find
  21. Yun Li, Zhe Liu, Yajing Kong, Guangrui Li, Jiyuan Zhang, Chao Bian, Feng Liu, Lina Yao, and Zhenbang Sun. 2025c. Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding. arXiv preprint arXiv:2501.16786. — [arXiv:2501.16786](https://arxiv.org/abs/2501.16786) find
  22. Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang W Chen. 2024a. E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding. Advances in Neural Information Processing Systems, 37:32076–32110. find
  23. Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024b. TempCompass: Do Video LLMs Really Understand Videos? Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772. find
  24. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021. Video Swin Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192–3201. find
  25. Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021. CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval. Neurocomputing, 508:293–304. find
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 35558 10 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730…
  27. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, and 1 others. 2023. Perception Test: A Diagnostic Benchmark for Multimodal Video Models. Advances in Neural Information Processing Systems, 36:42… find
  28. Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. find
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.
  31. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805.
  32. Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. 2024. Tarsier: Recipes for Training and Evaluating Large Video Description Models. arXiv preprint arXiv:2407.00634. — [arXiv:2407.00634](https://arxiv.org/abs/2407.00634) find
  33. Ning Wang, Guangming Zhu, Liang Zhang, Peiyi Shen, Hongsheng Li, and Cong Hua. 2021. Spatiotemporal interaction graph parsing networks for human-object interaction recognition. In Proceedings of the 29th ACM international conference on multimedia, pages 4985–4993. find
  34. Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. 2024. Can I Trust Your Answer? Visually Grounded Video Question Answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13204– 13214. find
  35. Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang, Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettlemoyer Christoph Feichtenhofer. 2021. VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. In Conference on Empirical Methods in Natural Language Processing. find
  36. Yibin Yan, Jilan Xu, Shangzhe Di, Yikun Liu, Yudi Shi, Qirui Chen, Zeqian Li, Yifei Huang, and Weidi Xie. 2025. Learning Streaming Video Representation via Multitask Training. arXiv preprint arXiv:2504.20041. — [arXiv:2504.20041](https://arxiv.org/abs/2504.20041) find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.
  38. En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Jingyu Wang, and 1 others. 2025. Unhackable Temporal Rewarding for Scalable Video MLLMs. arXiv preprint arXiv:2502.12081. — [arXiv:2502.12081](https://arxiv.org/abs/2502.12081) find
  39. Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. 2025a. Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning. arXiv preprint arXiv:2508.04416. — [arXiv:2508.04416](https://arxiv.org/abs/2508.04416) find
  40. Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. 2025b. TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning. arXiv preprint arXiv:2504.09641. — [arXiv:2504.09641](https://arxiv.org/abs/2504.09641) find
  41. Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. 2025a. VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning. arXiv preprint arXiv:2505.12434. — [arXiv:2505.12434](https://arxiv.org/abs/2505.12434) find
  42. Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. 2025b. TimeZero: Temporal Video Grounding with Reasoning-Guided LVLM. arXiv preprint arXiv:2503.13377. — [arXiv:2503.13377](https://arxiv.org/abs/2503.13377) find
  43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in neural information processing systems, 35:24824–24837. 35559 11

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-museg-reinforcing-video-temporal-understanding/