Elamur Research⌕
☰
home / papers / 2026 / Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding

Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding

Paper · 2026 · Ke Ma, Jiaqi Tang, Bin Guo et al.

Summary

In streaming video understanding, a Video-LLM sees footage frame by frame. In the proactive setting, it must also decide when it has seen enough to answer a question about events that may happen later. Earlier solutions take one of three routes. Some fine-tune on frame-level silence/response labels, which tend to be noisy. Others use query-agnostic thresholds on frame differences. A third group asks the model through prompting. All of them represent the visual evidence and the query's conditions only implicitly. Response-G1 makes both explicit as scene graphs. It works in three stages and needs no fine-tuning:

  1. A Video-LLM generates query-guided scene graphs from incoming clips.
  2. A memory bank stores these graphs and returns those closest to a graph form of the query.
  3. The retrieved graphs, with timestamps, are given to the Video-LLM, which uses a trigger prompt to decide per frame whether to stay silent or answer.

On StreamingBench and OVO-Bench, the authors report state-of-the-art results for both proactive and reactive tasks. The approach matters because it offers a training-free and more interpretable way to time responses in live assistants and embodied agents.

Key points

  • The method is evaluated on StreamingBench and OVO-Bench, with proactive subtasks PO, REC, SSR and CRR (Clues Reveal Responding).
  • Retrieval works better when the query is embedded as graph text rather than raw text. Scores rise from 42.4/40.2/69.3/56.0 to 44.0/41.9/71.1/61.7 on PO/REC/SSR/CRR.
  • Query-guided graph generation beats direct object injection ("Object-Guidance"). Injecting objects led the model to hallucinate triplets and respond too early.
  • Performance is fairly insensitive to the top-K retrieval size. K = 1 is enough for tasks centered on the latest frames, such as SSR and PO.
  • The case study compares against Qwen3-VL and TimeChat-Online on an OVO-Bench CRR example. Both baselines never respond, while Response-G1 answers at the right moment.
  • The authors list several limitations:
    • limited gains on "why"-style questions;
    • a fixed clip size;
    • a trade-off between query relevance and hallucination in open-set scene graph generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Northwestern Polytechnical UniversityXi'an · China13 works · 67 citations🎓Harbin Engineering UniversityHarbin · China2 works · 9 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu and Yunhao Liu. 2026. Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2042

@inproceedings{ma2026response,
  title = {{Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding}},
  author = {Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu and Yunhao Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2042},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2042}
}

References

48 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, and 1 others. 2025. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661. — [arXiv:2509.23661](https://arxiv.org/abs/2509.23661) find
  2. Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. ↗
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv prepri…
  4. Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024a. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti… find
  5. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, and 1 others. 2024b. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101.
  6. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and 1 others. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476. find
  7. Sanghyeok Chu, Seonguk Seo, and Bohyung Han. 2025. Fine-grained captioning of long videos through scene graph consolidation. In Forty-second International Conference on Machine Learning. find
  8. Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16372–16382. find
  9. Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. 2025. Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint arXiv:2503.00540. find
  10. Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. 2024. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023. — [arXiv:2408.14023](https://arxiv.org/abs/2408.14023) find
  11. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, and 1 others. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.
  12. Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. 2025. Online video understanding: Ovbench and videochat-online. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3328–3338. find
  13. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  14. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678. find
  15. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. decision, and reaction. In Proceedings of the Computer Vision and Pattern Recognitio…
  16. Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. 2024b. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 28076–28086. find
  17. Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. 2024. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:119336–119360. find
  18. Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. 2025. Lion-fs: Fast & slow video-language thinker as online video assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3240–3251. find
  19. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024a. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699.
  20. Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. 2024b. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628. — [arXiv:2411.03628](https://arxiv.org/abs/2411.03628) find
  21. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Lla-vanext: Improved reasoning, ocr, and world knowledge. find
  22. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024b. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542. — [arXiv:2408.15542](https://arxiv.org/abs/2408.15542) find
  23. Trong-Thuan Nguyen, Pha Nguyen, Jackson Cothren, Alper Yilmaz, and Khoa Luu. 2025. Hyperglm: Hypergraph for video scene graph generation and anticipation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29150– 29160. find
  24. Trong-Thuan Nguyen, Pha Nguyen, and Khoa Luu. 2024. Hig: Hierarchical interlacement graph approach to scene graph generation in video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18384–18394. find
  25. Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, and 1 others. 2025. Ovo-bench: How far is your video-llms from real-world online video understanding? In Proceedings of the Computer Vision and Pattern Recognition Conference… find
  26. Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video llms with active real-time interaction via disentangled perception, Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object… find
  27. Brigit Schroeder and Subarna Tripathi. 2020. Structured query-based image retrieval using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 178–179. find
  28. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, and 1 others. 2024. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434. — [arXiv:2410.17434](https://arxiv.org/abs/2410.17434) find
  29. Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, and Ying-Cong Chen. 2024. Hawk: Learning to understand open-world video anomalies. Advances in Neural Information Processing Systems, 37:139751–139785. find
  30. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  31. Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. 2025a. Streambridge: Turning your offline video large language model into a proactive streaming assistant. arXiv preprint arXiv:2505.05467. — [arXiv:2505.05467](https://arxiv.org/abs/2505.05467) find
  32. Hao Wang, Bin Guo, Mengqi Chen, Qiuyun Zhang, Yasan Ding, Ying Zhang, and Zhiwen Yu. 2025b. Cascade context-oriented spatio-temporal attention network for efficient and fine-grained video-grounded dialogues. Frontiers of Computer Science, 19(7):197329. find
  33. Hao Wang, Bin Guo, Yating Zeng, Mengqi Chen, Yasan Ding, Ying Zhang, Lina Yao, and Zhiwen Yu. 2025c. Enabling harmonious human-machine interaction with visual-context augmented dialogue system: A review. ACM Transactions on Information Systems, 43(3):1–59. find
  34. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  35. Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, and Xiaojie Jin. 2025a. Flash-vstream: Efficient real-time understanding for long video streams. arXiv preprint arXiv:2506.23825. — [arXiv:2506.23825](https://arxiv.org/abs/2506.23825) find
  36. Zichen Wen, Yiyu Wang, Chenfei Liao, Boxue Yang, Junxian Li, Weifeng Liu, Haocong He, Bolong Feng, Xuyang Liu, Yuanhuiyi Lyu, and 1 others. 2025. Ai for service: Proactive assistance with ai glasses. arXiv preprint arXiv:2510.14359. — [arXiv:2510.14359](https://arxiv.org/abs/2510.14359) find
  37. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852. find
  38. Junbin Xiao, Pan Zhou, Angela Yao, Yicong Li, Richang Hong, Shuicheng Yan, and Tat-Seng Chua. 2023. Contrastive video question answering via video graph transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13265–13280. find
  39. Yulin Zhang, Cheng Shi, Yang Wang, and Sibei Yang. 2025b. Eyes wide open: Ego proactive video-llm for streaming video. arXiv preprint arXiv:2510.14560. — [arXiv:2510.14560](https://arxiv.org/abs/2510.14560) find
  40. Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. 2025. Streaming video understanding and multi-round interaction with memory-enhanced knowledge. arXiv preprint arXiv:2501.13468. — [arXiv:2501.13468](https://arxiv.org/abs/2501.13468) find
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  42. Minghang Zheng, Yuxin Peng, Benyuan Sun, Yi Yang, and Yang Liu. 2025. Hierarchical event memory for accurate and low-latency online video temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21589– 21599. find
  43. Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. 2024. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18243–18252. find
  44. Dongil Yang, Minjin Kim, Sunghwan Mac Kim, Beong-woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, and Jinyoung Yeo. 2025b. Llm meets scene graph: Can large language models understand and generate scene graphs? a benchmark and empirical study. In Proceedings of the 63rd Annual Meeting of the Associ… find
  45. Haolin Yang, Feilong Tang, Lingxiao Zhao, Xiang An, Ming Hu, Huifa Li, Xinlin Zhuang, Yifan Lu, Xiaofeng Zhang, Abdalla Swikir, and 1 others. 2025c. Streamagent: Towards anticipatory agents for streaming video understanding. arXiv preprint arXiv:2508.01875. — [arXiv:2508.01875](https://arxiv.org/abs/2508.01875) find
  46. Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, and 1 others. 2025. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 10807–… find
  47. Sangwoong Yoon, Woo Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. 2021. Image-to-image retrieval by learning similarity between scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10718–10726. find
  48. Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, and 1 others. 2025. Streamforest: Efficient online video understanding with persistent event memory. arXiv preprint arXiv:2509.24871. — [arXiv:2509.24871](https://arxiv.org/abs/2509.24871) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-response-g1-explicit-scene-graph/