Elamur Research⌕
☰
home / papers / 2026 / MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding

MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding

Paper · 2026 · Jie Zhang, Qilang Ye, Hao Zhou et al.

Summary

Most text-to-video retrieval systems embed every video and compare each query against the whole collection. This is costly, an O(N) scan per query. It also handles poorly the gap between short, specific queries and visually rich videos. MAVIS addresses both problems with a training-free, multi-agent pipeline built on off-the-shelf MLLMs such as GPT-4o and Qwen3.

First, a Description Agent converts each video into a text-based Structured Semantic Library, indexed by Scene, Object and Action. At query time, a planner splits the request into these atomic aspects and activates only the relevant specialist agents. Each agent proposes candidates on its own. The agents then hold a Logic-aware Debate, in which any one of them can veto a video that contradicts its dimension. Only the remaining disputed candidates go to a fine-grained visual–text Matching Tool for final ranking.

On MSR-VTT, MSVD and ActivityNet, the authors report results competitive with, and in places better than, fine-tuned methods. They present this as a scalable and interpretable alternative to dual-encoder retrieval.

Key points

  • The pipeline has three stages: Structured Video Understanding, Multi-agent Collaboration (planner, independent proposals, veto debate), and Fine-grained Matching.
  • The system needs no task-specific fine-tuning and is evaluated on MSR-VTT, MSVD and ActivityNet.
  • In the analysis of candidate-set size, MAVIS reaches a peak of 78.6% R@1 while passing only 16 candidates to fine-grained matching.
  • Threshold ablation:
    • A lenient proposal threshold τ_soft = 0.3 slightly raises recall but inflates the candidate pool to about 42.1 videos; 0.5 is the chosen setting.
    • Setting the veto threshold τ_hard too high discards true positives, while setting it too low lets distractors through.
  • In case studies, the authors contrast MAVIS with InternVideo2 (global embeddings, prone to false positives) and Merlin (a single-agent MLLM, prone to tunnel vision and hallucination).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations🎓Nankai UniversityTianjin · China24 works · 762 citations🎓Great Bay UniversityDongguan · China3 works · 3 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Jie Zhang, Qilang Ye, Hao Zhou, Haochen Liang and Fei Luo. 2026. MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1094

@inproceedings{zhang2026mavis,
  title = {{MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding}},
  author = {Jie Zhang, Qilang Ye, Hao Zhou, Haochen Liang and Fei Luo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1094},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1094}
}

References

45 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang. 2025. Lvagent: Long video understanding by multi-round dynamical collaboration of mllm agents. arXiv preprint arXiv:2503.10200. — [arXiv:2503.10200](https://arxiv.org/abs/2503.10200) find
  2. Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. 2023. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842–72866. find
  3. Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang, Shujie Chen, Xirong Li, and Xun Wang. 2022. Partially relevant video retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 246–257. find
  4. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
  5. Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. 2024. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75–92. Springer. find
  6. Sheng Fang, Tiantian Dang, Shuhui Wang, and Qingming Huang. 2024. Linguistic hallucination for text-based video retrieval. IEEE Transactions on Circuits and Systems for Video Technology. find
  7. Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2013. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In Proceedings of the IEEE international c… find
  8. Donghoon Han, Eunhwan Park, Gisang Lee, Adam Lee, and Nojun Kwak. 2024. Merlin: Multimodal embedding refinement via llm-based iterative navigation for text-video retrieval-rerank pipeline. arXiv preprint arXiv:2407.12508. find
  9. Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. 2023. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 938–946. find
  10. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706–715. find
  11. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  12. Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer. find
  13. Yang Liu, Shudong Huang, Deng Xiong, and Jiancheng Lv. 2025. Learning dynamic similarity by bidirectional hierarchical sliding semantic probe for efficient text video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5667–5675. find
  14. Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293– 304. find
  15. Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. 2024. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093. find
  16. Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM international conference on multimedia, pages 638–647. find
  17. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  18. Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313–14323. find
  19. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154–38180.
  20. Haoran Tang, Meng Cao, Jinfa Huang, Ruyang Liu, Peng Jin, Ge Li, and Xiaodan Liang. 2025. Muse: Mamba is efficient multi-scale learner for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7238–7246. find
  21. Kaibin Tian, Ruixiang Zhao, Zijie Xin, Bangxiang Lan, and Xirong Li. 2024. Holistic features are almost sufficient for text-to-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17138–17147. find
  22. Twelve Labs. 2024. Introducing Marengo 2.6: A state-of-the-art video foundation model for any-to-any search. https://www.twelvelabs.io/blog/ introducing-marengo-2-6. Accessed: 2026-01-05. ↗
  23. Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. 2024. Text is mass: Modeling as stochastic embedding for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16551–16560. find
  24. Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. 2023a. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14549–14560. find
  25. Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, and 1 others. 2025a. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. In Proceedings of the Computer Vision and Patter… find
  26. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, and Yaohui Wang. 2023b. Internvid: A large-scale video-text dataset for multimodal understanding and generation. find
  27. Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, and Yansong Shi. 2025b. Internvideo2: Scaling foundation models formultimodal video understanding. In European Conference on Computer Vision. find
  28. Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2023. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23034–23044. find
  29. Peng Wu, Wanshun Su, Xiangteng He, Peng Wang, and Yanning Zhang. 2025. Varcmp: Adapting cross-modal pre-training models for video anomaly retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8423–8431. of the AAAI Conference on Artificial Intelligence, volume… find
  30. Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2023. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36:76749–76771. find
  31. Jiayu Zhang, Pengjie Tang, Yunlan Tan, and Hanli Wang. 2025. Mgtr-miss: More ground truth retrieving based multimodal interaction and semantic supervision for video description. Neural Networks, page 107817. find
  32. Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. 2023. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10704–10713. find
  33. Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, and Zitong Yu. 2026. Retrieving to recover: Towards incomplete audio-visual question answering via semantic-consistent purification. Preprint, arXiv:2604.10695.
  34. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296. find
  35. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339.
  36. Xiangpeng Yang, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2024. Dgl: Dynamic global-local prompt tuning for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6540–6548. find
  37. Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, and Yue Zhao. 2024. Videoprism: A foundational visual encoder for video understanding. find
  38. Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. 2025. Vca: Video curious agent for long video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20168–20179. find
  39. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, and 1 others. 2023. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852. — [arXiv:2310.01852](https://arxiv.org/abs/2310.01852) find
  40. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  41. Qilang Ye, Zitong Yu, Rui Shao, Yawen Cui, Xiangui Kang, Xin Liu, Philip Torr, and Xiaochun Cao. 2025. Cat+: Investigating and enhancing audio-visual understanding in large language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  42. Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. 2024. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In European Conference on Computer Vision, pages 146–164. Springer. find
  43. Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, and Yu Zhou. 2026a. When eyes and ears disagree: Can mllms discern audio-visual confusion? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11955–11963. find
  44. Qilang Ye, Yu Zhou, Lian He, Jie Zhang, Xuanming Guo, Jiayu Zhang, Mingkui Tan, Weicheng Xie, Yue Sun, Tao Tan, and 1 others. 2026b. Sugar: Learning skeleton representation with visual-motion knowledge for action recognition. In Proceedings Yijie Zhu, Jie He, Rui Shao, Kaishen Yuan, Tao Tan, Xiaoche… — [arXiv:2603.08361](https://arxiv.org/abs/2603.08361) find
  45. Yijie Zhu, Rui Shao, Ziyang Liu, Jie He, Jizhihui Liu, Jiuru Wang, and Zitong Yu. 2026b. H-gar: A hierarchical interaction framework via goal-driven observation-action refinement for robotic manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-mavis-multi-agent-video-retrieval/