Elamur Research⌕
☰
home / papers / 2026 / Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction

Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction

Paper · 2026 · Galann Pennec, Zhengyuan Liu, Nicholas Asher et al.

Summary

Vision-Language Models can now take in long videos, but running them over a full movie is expensive, and they often overlook important visual details. The authors treat this as a selection problem. They aim to find the short stretches of video whose visual content matters for a summary and cannot be recovered from the dialogue alone. Their pipeline works as follows:

  • Split the video into 20-second clips.
  • Caption each clip with a small model (Qwen2.5-Omni-7B).
  • Have an LLM pick the top K clips, zero-shot or two-shot.
  • Re-describe the chosen clips with a stronger model (Gemini 2.5 Flash-Lite) and merge these descriptions into the timestamped transcript, forming a screenplay-like document.
  • Summarize that document with Gemini 2.5 Flash.

For evaluation, the authors derive reference clips from the human screenplays and summaries in MovieSum. These references cover under 6% of a film's runtime yet are enough to produce a full multimodal summary. Summaries built from the selected clips come close to those built from the references and capture clearly more visual information than random selection, while keeping compute costs modest.

Key points

  • Evaluation data: experiments use the MovieSum test split, which contains 200 of the dataset's 2200 films. Movies average about two hours, with summaries of roughly 635 words and screenplays of about 25K words.
  • Metrics: quality is measured with ROUGE, METEOR, and MFactSum. MFactSum averages two recall scores: the share of visual facts and the share of dialogue-derived facts from the gold summary that the output covers.
  • Recall of reference clips: with Qwen2.5-Omni-7B captions, zero-shot selection reaches R@25/50/75 of 11.89/19.89/26.60. Two-shot selection reaches 13.82/21.95/26.25.
  • Scene segmentation: replacing fixed clips with scene-based segments (average scene length 73 seconds, further split into ~20-second pieces) did not improve selection.
  • Subjectivity check: the authors collected 54 alternative summaries from The Movie Spoiler. Clips derived from them recover about 48.8% of the MovieSum reference clips, which shows that what counts as salient varies with the summary source.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSummarization · 232 works

Cite

Cite as Galann Pennec, Zhengyuan Liu, Nicholas Asher, Philippe Muller and Nancy Chen. 2026. Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pennec2026minimal,
  title = {{Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction}},
  author = {Galann Pennec, Zhengyuan Liu, Nicholas Asher, Philippe Muller and Nancy Chen},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.22/}
}

References

65 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yash Kumar Atri, Shraman Pramanick, Vikram Goyal, and Tanmoy Chakraborty. 2021. See, hear, read: Leveraging multimodality with guided attention for abstractive text summarization. Knowl. Based Syst., 227:107152. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang,… — Qwen2.5-VL Technical Report
  3. Hénaff. 2024. Memory consolidation enables long-context video understanding. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  4. Brian Y. Chen, Xiangyuan Zhao, and Yingnan Zhu. 2024. Personalized video summarization by multimodal video understanding. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, pages 4382–4389. ACM. find
  5. Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106– 4118, Online. Association for Computational Ling…
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, Krishna Har… — Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
  7. Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. A survey on dialogue summarization: Recent advances and new frontiers. arXiv preprint arXiv:2107.03175. — [arXiv:2107.03175](https://arxiv.org/abs/2107.03175) find
  8. Ivana Balazevic, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J. Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng,… — Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  9. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  10. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. MA-LMM: memory-augmented large multimodal model for long-term video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, Jun… find
  11. De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, and Jan Kautz. 2025. FRAG: frame selection augmented generation for long video and long document understanding. CoRR, abs/2504.17447. — [arXiv:2504.17447](https://arxiv.org/abs/2504.17447) find
  12. Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S. Ryoo. 2024. Language repository for long video understanding. CoRR, abs/2403.14622. find
  13. Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  14. Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zisserman, and Federico Tombari. 2024. Text-conditioned resampler for long form video understanding. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXVI, volume 15144 of Le… find
  15. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 706–715. IEEE Computer Society. find
  16. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 9332–9346.
  17. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. 20… find
  18. Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2017. Multi-modal summarization for asynchronous collection of text, image, audio and video. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1092–1102, Copenhagen, Denmark. Association… find
  19. Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024. LLaMA-VID: An image is worth 2 tokens in large language models. In Computer Vision - ECCV 2024 -18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XLVI, volume 15104 of Lecture Notes in Computer Science, pages 323–34… find
  20. Dongqi Liu, Chenxi Whitehouse, Xi Yu, Louis Mahon, Rohit Saxena, Zheng Zhao, Yifu Qiu, Mirella Lapata, and Vera Demberg. 2025. What is that talk about? A video-to-text summarization dataset for scientific presentations. In Proceedings of the 63rd Annual Meeting of the Association for Computational L…
  21. Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 92–106, Online and Punta Cana, Dominican Republic. Association for Computational Lingui… find
  22. Zhengyuan Liu and Nancy Chen. 2022. Dynamic sliding window modeling for abstractive meeting summarization. In Proc. find
  23. Interspeech 2022, pages 5150– 5154. find
  24. Zhengyuan Liu, Angela Ng, Sheldon Lee, Ai Ti Aw, and Nancy F Chen. 2019. Topic-aware pointer-generator networks for summarizing spoken conversations. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 814–821. IEEE. find
  25. Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. Coreference-aware dialogue summarization. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509–519, Singapore and Online. Association for Computational Linguistics.
  26. Louis Mahon and Mirella Lapata. 2024a. A modular approach for multimodal summarization of TV shows. find
  27. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 8272–8291. Association for Computational Linguistics. find
  28. Louis Mahon and Mirella Lapata. 2024b. Screenwriter: Automatic screenplay generation and movie summarisation. CoRR, abs/2410.19809. — [arXiv:2410.19809](https://arxiv.org/abs/2410.19809) find
  29. Louis Mahon and Mirella Lapata. 2025. Parameter-free video segmentation for vision and language understanding. CoRR, abs/2503.01201. — [arXiv:2503.01201](https://arxiv.org/abs/2503.01201) find
  30. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,… find
  31. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), Octo… find
  32. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computation…
  33. Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. 2021. CLIP-It! language-guided video summarization. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 13988–14000. find
  34. Taichi Nishimura, Shota Nakada, and Masayoshi Kondo. 2024. On the audio hallucinations in large audio-video language models. CoRR, abs/2401.09774. — [arXiv:2401.09774](https://arxiv.org/abs/2401.09774) find
  35. OpenAI. 2024. Hello GPT-4o. Accessed: 2024-11-6. find
  36. Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2021. Movie summarization via sparse graph construction. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on… find
  37. Pinelopi Papalampidi and Mirella Lapata. 2023. Hierarchical3D adapters for long video-to-text summarization. In Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1267–1290. Association for Computational Linguistics. find
  38. Jean Park, Kuk Jin Jang, Basam Alasaly, Sriharsha Mopidevi, Andrew Zolensky, Eric Eaton, Insup Lee, and Kevin Johnson. 2024a. Assessing modality bias in video question answering benchmarks with multimodal large language models. CoRR, abs/2408.12763. — [arXiv:2408.12763](https://arxiv.org/abs/2408.12763) find
  39. Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryoo, Donghyun Kim, and Michael S. Ryoo. 2024b. Too many frames, not all useful: Efficient strategies for long-form video QA. CoRR, abs/2406.09396.
  40. Galann Pennec, Zhengyuan Liu, Nicholas Asher, Philippe Muller, and Nancy F. Chen. 2025. Integrating video and text: A balanced approach to multimodal summary generation and evaluation. CoRR, abs/2505.06594. find
  41. Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. 2024. Streaming long video understanding with large language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vanco… find
  42. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  43. Rohit Saxena and Frank Keller. 2024. MovieSum: An abstractive summarization dataset for movie screenplays. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 4043– 4050. Association for Computational Linguistics. find
  44. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. find
  45. Yuhan Shen, Linjie Yang, Longyin Wen, Haichao Yu, Ehsan Elhamifar, and Heng Wang. 2024. Exploring the role of audio in video captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Workshops, Seattle, WA, USA, June 17-18, 2024, pages 2090–2100. IEEE. find
  46. Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. 2025. VideoRoPE: What makes for good video rotary position embedding? CoRR, abs/2502.05173. — [arXiv:2502.05173](https://arxiv.org/abs/2502.05173) find
  47. Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. 2024. LongVLM: Efficient long video understanding via large language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXXIII, volume 15091 of Lecture… find
  48. Aditya Kumar Singh, Dhruv Srivastava, and Makarand Tapaswi. 2024. "previously on..." from recaps to story summarization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13635–13646. IEEE. find
  49. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. 2024. MovieChat: From dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer Vision and Patte… find
  50. Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023. In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 56–67, Singapore. Association… find
  51. Bin Wang, Zhengyuan Liu, and Nancy Chen. 2023. Instructive dialogue summarization with query aggregations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7630–7653, Singapore. Association for Computational Linguistics. find
  52. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2024a. LVBench: An extreme long video understanding benchmark. CoRR, abs/2406.08035. find
  53. Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. 2025. AdaReTaKe: Adaptive redundancy reduction to perceive longer for video-language understanding. CoRR, abs/2503.12559. find
  54. Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2024b. VideoAgent: Long-form video understanding with large language model as agent. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXX, volume 15138 of Lecture Not… find
  55. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2024c. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. CoRR, abs/2405.19209. find
  56. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. CoRR, abs/2503.20215.
  57. Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. 2024. LongVILA: Scaling long-context visual language models for long videos. CoRR, a… find
  58. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find
  59. Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. 2024b. VCA: video curious agent for long video understanding. CoRR, abs/2412.10471. — [arXiv:2412.10471](https://arxiv.org/abs/2412.10471) find
  60. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI conference on artificial intelligen… find
  61. Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2023. Self-chained image-language model for video localization and question answering. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA… find
  62. Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun. 2025. Frame-Voyager: Learning to query frames for video large language models. In The Thirteenth International Conference on Learning Represen… find
  63. NB_CAPTIONS. Caption caption_number: Justification why the Caption describes crucial action for the summary Yifan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2024b. Debiasing multimodal large language models. CoRR, abs/2403.05262. — [arXiv:2403.05262](https://arxiv.org/abs/2403.05262) find
  64. Zhuo Zhi, Qiangqiang Wu, Minghe shen, Wenbo Li, Yinchuan Li, Kun Shao, and Kaiwen Zhou. 2025. VideoAgent2: Enhancing the LLM-based agent system for long-form video understanding by uncertainty-aware CoT. CoRR, abs/2504.04471. — [arXiv:2504.04471](https://arxiv.org/abs/2504.04471) find
  65. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. MLVU: A comprehensive benchmark for multi-task long video understanding. CoRR, abs/2406.04264. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pennec-2026-minimal-clips-maximum-salience-long/