Elamur Research⌕
☰
home / papers / 2024 / LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

Paper · 2024 · Haoning Wu, Dongxu Li, Bei Chen et al.

Summary

Benchmarks for long-context models are mostly text-only. Existing video QA sets also suffer from single-frame bias: models score about the same whether they see a few frames or many. LongVideoBench addresses this with a multiple-choice benchmark of web-collected, subtitled videos lasting up to an hour, in which visual frames and subtitles are interleaved. Its core task, referring reasoning, works in two steps. Each question first describes a referring query that points to one or more moments of the video (the referred context), and then asks about details there. Answering requires the model to locate those moments in a long input and then reason over them. Questions come at two levels: perception of a single moment, and relations across several moments. Proprietary models such as GPT-4o, Gemini-1.5-Pro and GPT-4-Turbo find the benchmark hard, and open-source models trail them further. Accuracy rises only for models that can actually use more frames, which makes the benchmark a useful yardstick for future long-context LMMs.

Key points

  • Scale and comparison: 3,763 videos and 6,678 human-annotated multiple-choice questions. The test answers are hidden to limit contamination.
  • Video coverage: themes include movies, news, life and knowledge. Videos fall into four duration bins: 8–15 s, 15–60 s, 3–10 min and 15–60 min. The average length is 473 s, longer than EgoSchema or MovieChat-1K.
  • Question categories: 17 fine-grained categories split into two levels:
    • Perception (L1): 3,204 questions.
    • Relation (L2): 3,474 questions.
    • L2 is harder overall. Sequence of Scenes (SSS), where distractors are reorderings of the true scene order, is the weakest category for every model.
  • Open-source models: PLLaVA-34B ranks highest. Larger PLLaVA variants (13B, 34B) gain 5.9% and 14.3% over a smaller counterpart trained on the same data, indicating that scaling the language model helps.
  • Effect of moment position: accuracy falls when the referred moment sits near the start or middle of the video, and the drop grows with video length. This mirrors needle-in-a-haystack findings for text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksVision & language · 811 works

Cite

Cite as Haoning Wu, Dongxu Li, Bei Chen and Junnan Li. 2024. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. arXiv:2407.15754.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.15754 · arXiv 2407.15754

@misc{wu2024longvideobench,
  title = {{LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding}},
  author = {Haoning Wu, Dongxu Li, Bei Chen and Junnan Li},
  year = {2024},
  doi = {10.48550/arxiv.2407.15754},
  eprint = {2407.15754},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.15754}
}

References

39 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
  2. OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, May 2024a. ↗
  3. Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  4. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. — RULER: What's the Real Context Size of Your Long-Context Language Models?
  5. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks, 2024a. gkamradt. Llmtest_needleinahaystack, 2024. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack. ↗
  6. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017. find
  7. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. arXiv preprint arXiv:1906.02467, 2019. URL https://arxiv.org/abs/1906.02467. ↗
  8. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions, 2021. find
  9. Gaoang Wang, Jenq-Neng Hwang, Yanting Zhang, and Yan Lu. Mv-bench: A benchmark for multi-view video understanding. arXiv preprint arXiv:2308.12345, 2023. URL https://arxiv.org/abs/2308.12345. — [arXiv:2308.12345](https://arxiv.org/abs/2308.12345) ↗
  10. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/90ce332aff156b910b002ce4e6880dec-Ab… ↗
  11. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023. URL https://arxiv.… ↗
  12. Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhenhua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.04817, 2023a. URL https://arxiv.org/pdf/2312.04817. — [arXiv:2312.04817](https://arxiv.org/abs/2312.04817) ↗
  13. OpenAI. Whisper-v3-large: A large scale multimodal model. https://github.com/openai/whisper-v3-large, 2024b. ↗
  14. Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtao Zhai, and Weisi Lin. Q-align: Teaching lmms for visual scoring via discrete text-defined levels. In ICML2024, 2024. find
  15. Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2: Scaling video foundation models for multimodal video understanding, 20… find
  16. Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding, 2023b.
  17. Qiang Zhang, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023c. URL https://arxiv.org/abs/2306.05424. ↗
  18. Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024. find
  19. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2023a.
  20. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
  21. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744, 2023a.
  22. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. CoRR, abs/2311.04257, 2023.
  23. Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023c.
  24. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding, 2024. find
  25. Reuben Tan, Ximeng Sun, Ping Hu, Jui hsien Wang, Hanieh Deilamsalehy, Bryan A. Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm, 2024. find
  26. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2024a.
  27. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Alli… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  28. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
  29. Huggingface. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023. URL https://huggingface.co/blog/idefics. ↗
  30. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  31. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, 2024.
  32. SkunkworksAI. BakLLaVA, 2024. URL https://github.com/SkunkworksAI/BakLLaVA. ↗
  33. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. CoRR, abs/2305.06500, 2023.
  34. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, abs/2304.08485, 2023b.
  35. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Lava-next: Improved reasoning, ocr, and world knowledge, 2024b. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/. ↗
  36. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023.
  37. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
  38. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions, 2024. URL https://arxiv. org/abs/2406.04325… — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) find
  39. Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024. URL https://llava-vl.github.io/blog/2024-04-30-llava-next-video/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2024-longvideobench-benchmark-long-context-interleaved/