Elamur Research⌕
☰
home / papers / 2026 / Agentic Very Long Video Understanding

Agentic Very Long Video Understanding

Paper · 2026 · Aniket Rege, Arka Sadhu, Yuliang Li et al.

Summary

Wearable assistants such as smart glasses would need to answer questions about what a user experienced over days or weeks of first-person footage. Current LLM and retrieval-based approaches struggle here because context windows are limited and plain retrieval cannot chain facts across days or modalities. The authors propose EGAgent, a tool-using agent built around a time-aware entity scene graph. Nodes are people, objects and places. Edges are relations such as talks-to or interacts-with, each tagged with the time spans when it holds. Because of these time tags, the graph can be extended incrementally as new footage arrives. A planning agent splits a query into subtasks and combines three tools: graph search, a hybrid SQL/semantic frame search, and transcript search. On EgoLifeQA, a week-long egocentric benchmark with more than 50 hours of video, EgoLifeQA accuracy reaches a new best of 57.5%. On Video-MME (Long), EgoLifeQA is competitive at 74.1% while using far fewer frames than most rivals. The work suggests that structured, temporally grounded memory is a practical route to multi-day video reasoning.

Key points

  • Benchmarks:
    • EgoLife contains continuous, week-long video from six wearers.
    • Evaluation used EgoLifeQA (multiple-choice) and Video-MME (Long).
  • EgoLifeQA results:
    • The best configuration (Gemini 2.5 Pro, graph extracted from transcripts plus captions, LLM-based transcript search) scores 57.5%, versus 36.2% for EgoButler with GPT-4o.
    • Gains over the prior state of the art are largest on the multi-hop categories: +32% on RelationMap and +39.7% on TaskMaster.
  • Video-MME (Long) results:
    • EGAgent with Gemini 2.5 Pro reaches 74.1% while retrieving only 50 frames.
    • Only native Gemini 2.5 Pro scores higher, at 82.0% with 256 uniformly sampled frames.
    • EGAgent with Qwen2.5-VL-7B (47.8%) matches AdaVideoRAG (47.7%) using roughly 15 times fewer frames (50 vs. 768).
  • Ablations:
    • The entity graph raises temporal-localization recall@10s from 0.232 to 0.884.
    • Adding visual captions to graph extraction gives about +2.6% accuracy.
    • Replacing LLM transcript search with BM25 cuts token use 3.3×, at a cost of about 6.8% accuracy.
  • Cost and limitations:
    • Answering one question takes roughly 2–3 minutes.
    • Results depend on upstream perception quality and on the manually annotated speaker diarization available for EgoLife.
    • Code is released at github.com/facebookresearch/egagent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Aniket Rege, Arka Sadhu, Yuliang Li, Kejie Li, Ramya Korlakai Vinayak, Yuning Chai, Yong Jae Lee and Hyo Jin Kim. 2026. Agentic Very Long Video Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2161

@inproceedings{rege2026agentic,
  title = {{Agentic Very Long Video Understanding}},
  author = {Aniket Rege, Arka Sadhu, Yuliang Li, Kejie Li, Ramya Korlakai Vinayak, Yuning Chai, Yong Jae Lee and Hyo Jin Kim},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2161},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2161}
}

References

55 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In ICCV. find
  2. Shyamal Buch, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. 2025. Flexible frame selection for efficient video reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29071–29082. find
  3. Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang. 2025. Lvagent: Long video understanding by multi-round dynamical collaboration of mllm agents. arXiv preprint arXiv:2503.10200. — [arXiv:2503.10200](https://arxiv.org/abs/2503.10200) find
  4. Meng Chu, Yicong Li, and Tat-Seng Chua. 2025. Graphvideoagent: Enhancing long-form video understanding with entity relation graphs. In Proceed-License: Video-MME https://github.com/MME-Benchmarks/ings of the 33rd ACM International Conference on Multimedia, pages 4639–4648. ↗
  5. Anxhelo Diko, Tinghuai Wang, Wassim Swaileh, Shiyan Sun, and Ioannis Patras. 2025. Rewind: Understanding long videos with instructed learnable memory. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13734– 13743. find
  6. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753.
  7. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  8. Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Ar-jang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, and 1 others. 2023. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561. — [arXiv:2308.13561](https://arxiv.org/abs/2308.13561) find
  9. Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. 2024. Videoagent: A memory-augmented multimodal agent for video understanding. In ECCV, pages 75–92. Springer. find
  10. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. In ICLR. find
  11. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, pages 24108–24118.
  12. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  13. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, and 1 others. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012. find
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  15. Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779.
  16. Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, and 1 others. 2025. M-llm based video frame selection for efficient video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13702–13712. find
  17. Zeyi Huang, Yuyang Ji, Xiaofang Wang, Nikhil Mehta, Tong Xiao, Donghyun Lee, Sigmund Vanvalken-burgh, Shengxin Zha, Bolin Lai, Licheng Yu, and 1 others. 2025. Building a mind palace: Structuring environment-grounded semantic graphs for effective long video analysis with llms. In Proceedings of the C… find
  18. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325.
  19. AI LangChain. 2025. LangGraph: Building language agents as graphs. https://github.com/ langchain-ai/langgraph. Accessed: 2025-11-01. ↗
  20. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS, 33:9459–9474.
  21. Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian. 2023. Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction. arXiv preprint arXiv:2310.15556. find
  22. Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. 2025a. Bolt: Boost large vision-language model without training for long-form video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3318–3327. find
  23. Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. 2025b. Hybrid-level instruction injection for video token compression in multi-modal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 85… find
  24. Yujie Lu, Yale Song, William Wang, Lorenzo Torresani, and Tushar Nagarajan. 2025a. Vited: Video temporal evidence distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8501–8511. find
  25. Zijia Lu, ASM Iftekhar, Gaurav Mittal, Tianjian Meng, Xiawei Wang, Cheng Zhao, Rohith Kukkala, Ehsan Elhamifar, and Mei Chen. 2025b. Decafnet: Delegate and conquer for efficient temporal grounding in long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24066–… find
  26. Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. 2025. Video-RAG: Visually-aligned retrieval-augmented long video comprehension. In NeurIPS. find
  27. Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. 2025. Drvideo: Document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18936– 18946. find
  28. Sameer Malik, Ayush Singh, Moyuru Yamada, and Dis-hank Aggarwal. 2026. Ravu: Retrieval augmented video understanding with compositional reasoning over graph. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2869–2878. find
  29. Yuanbin Man, Ying Huang, Chengming Zhang, Bingzhe Li, Wei Niu, and Miao Yin. 2025. Adacmˆ 2: On understanding extremely long-term video with adaptive cross-modality memory reduction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8534–8544. find
  30. Arun Reddy, Alexander Martin, Eugene Yang, Andrew Yates, Kate Sanders, Kenton Murray, Reno Kriz, Celso M de Melo, Benjamin Van Durme, and Rama Chellappa. 2025. Video-colbert: Contextualized late interaction for text-to-video retrieval. In Proceedings of the Computer Vision and Pattern Recognition Co… find
  31. Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. 2025. Videorag: Retrieval-augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549. — [arXiv:2502.01549](https://arxiv.org/abs/2502.01549) find
  32. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  33. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, and 1 others. 2025. Longvu: Spatiotemporal adaptive compression for long video-language understanding. ICML. find
  34. Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. 2025. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169. find
  35. Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. 2025. Adaptive keyframe sampling for long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29118–29128. find
  36. Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, and Ziwei Liu. 2025. Ego-r1: Chain-of-tool-thought for ultra-long egocentric video reasoning. arXiv preprint arXiv:2506.13654. — [arXiv:2506.13654](https://arxiv.org/abs/2506.13654) find
  37. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others. 2025. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense feat… find
  38. David Wan, Han Wang, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. 2025. Clamr: Contextualized late-interaction for multimodal content retrieval. arXiv preprint arXiv:2506.06144. find
  39. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, and 1 others. 2025a. Lvbench: An extreme long video understanding benchmark. In CVPR, pages 22958–22967. find
  40. Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2024. Videoagent: Long-form video understanding with large language model as agent. In ECCV, pages 58–76. Springer. find
  41. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2025b. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3272–3283. find
  42. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. ArXiv, abs/2407.15754. find
  43. Chi Xu. 2025. Designing xreal one pro: the next generation of ost glasses. In SPIE AR, VR, MR Invited Talks 2025, volume 13415, page 1341504. SPIE. find
  44. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296. find
  45. Zhucun Xue, Jiangning Zhang, Xurong Xie, Yong Liu, Xiangtai Li, Dacheng Tao, and 1 others. 2025. Adavideorag: Omni-contextual adaptive retrieval-augmented efficient long video understanding. In NeurIPS. video processing in large vision-language models. In Proceedings of the Computer Vision and Patte… find
  46. Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, and 1 others. 2025b. Egolife: Towards egocentric life assistant. In CVPR, pages 28885–28900. find
  47. Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, and 1 others. 2025. Re-thinking temporal search for long-form video understanding. In Proceedings of the Computer Vision and Pattern Recogniti… find
  48. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and 1 others. 2025. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In ICLR. find
  49. Huaying Yuan, Zheng Liu, Junjie Zhou, Ji-Rong Wen, and Zhicheng Dou. 2025. Videodeepresearch: Long video understanding with agentic tool using. arXiv preprint arXiv:2506.10821. — [arXiv:2506.10821](https://arxiv.org/abs/2506.10821) find
  50. Ce Zhang, Yan-Bo Lin, Ziyang Wang, Mohit Bansal, and Gedas Bertasius. 2026. SiLVR: A simple language-based video reasoning framework. Transactions on Machine Learning Research. find
  51. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. find
  52. Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. 2025. Accelerating multimodal large language models by searching optimal vision token reduction. In Proceedings of the Computer Vision and Pattern Recognition… find
  53. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024a. Mlvu: Benchmarking multi-task long video understanding. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13691–13701. find
  54. Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. 2024b. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18243–18252. find
  55. Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. 2025a. Pvc: Progressive visual token compression for unified image and \#\# Acknowledgements This work was supported in part by NSF IIS2404180 and Institute of Information & communica… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rege-2026-agentic-very-long-video-understanding/