Elamur Research⌕
☰
home / papers / 2026 / The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning

The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning

Paper · 2026 · Mingkai Tian, Guorong Li, Yuankai Qi et al.

Summary

Zero-shot video captioning means describing videos without training on paired video-text data. Current text-only approaches retrieve textual prompts through CLIP at a single level of detail, such as noun phrases or full sentences. As a result, their captions often capture one salient element and miss interactions or context. The authors instead feed the decoder cues at three levels of granularity, drawn from separate memory banks:

  • frequent noun phrases;
  • scene-graph triples whose nodes are enriched with attribute-bearing noun phrases, chosen by BGE similarity to the source caption;
  • complete training captions.

During training, prompts are retrieved with noise-perturbed text embeddings and a decoder learns to reconstruct the caption, as in MultiCapCLIP. At inference, a category-aware retrieval step reduces redundancy among prompts while keeping them visually grounded. It groups noun phrases (classified with GPT-4) and scene graphs into categories, uses corpus priors, and applies top-p filtering.

On MSR-VTT, MSVD and VATEX, the method beats prior zero-shot systems in both in-domain and cross-domain tests. This suggests that explicitly modeling scene structure matters for this task.

Key points

  • With the same backbone, CIDEr improves over the previous best by 5.7% on MSR-VTT, 16.2% on MSVD and 3.4% on VATEX.
  • In the MSR-VTT⇒MSVD transfer setting, replacing plain top-K retrieval with category-aware retrieval plus top-p filtering has two effects:
    • CIDEr rises from 34.7 to 41.4;
    • prompt redundancy falls, with Self-BLEU dropping from 0.598 to 0.475.
  • Stronger encoders help considerably. Moving from CLIP (ViT/B-16) to GME-Qwen2VL-7B lifts CIDEr from 39.3 to 48.2 on MSR-VTT and from 41.4 to 62.2 on VATEX.
  • Building the prompts adds about 2 ms per video on MSVD. Video encoding takes about 97 ms by comparison.
  • The noun phrase and scene graph banks are truncated by frequency, so rare concepts may be underrepresented. The authors plan to use an information-gain criterion for selecting bank entries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Mingkai Tian, Guorong Li, Yuankai Qi, Anton van den Hengel and Qingming Huang. 2026. The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.98

@inproceedings{tian2026devil,
  title = {{The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning}},
  author = {Mingkai Tian, Guorong Li, Yuankai Qi, Anton {van den Hengel} and Qingming Huang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.98},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.98}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In IEEvalua-tion@ACL, pages 65–72.
  2. Qianyue Bao, Fang Liu, Licheng Jiao, Yang Liu, Shuo Li, Lingling Li, Xu Liu, Puhua Chen, and Wenping Ma. 2025. Erfc: Energy-aware reinforcement feedback calibration for zero-shot captioning. IEEE Trans. Circuits Syst. Video Technol. find
  3. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190–200.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186.
  5. Junjie Fei, Teng Wang, Jinrui Zhang, Zhenyu He, Chengjie Wang, and Feng Zheng. 2023. Transferable decoding with visual entities for zero-shot image captioning. In ICCV, pages 3113–3123. find
  6. Lianli Gao, Yu Lei, Pengpeng Zeng, Jingkuan Song, Meng Wang, and Heng Tao Shen. 2022. Hierarchical representation network with auxiliary tasks for video captioning and video question answering. IEEE Trans. Image Process., 31:202–215. find
  7. Xingwei He. 2021. Parallel refinements for lexically constrained text generation with BART. In EMNLP, pages 8653–8666. find
  8. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, pages 7514–7528.
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  10. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR.
  11. Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In ECCV, pages 709–727. find
  12. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis…
  13. Soeun Lee, Si-Woo Kim, Taewhan Kim, and Dong-Jin Kim. 2024. Ifcap: Image-like retrieval and frequency-based entity filtering for zero-shot captioning. In EMNLP, pages 20715–20727. find
  14. Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. 2023a. Decap: Decoding CLIP latents for zero-shot captioning via text-only training. In ICLR. find
  15. Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. 2023b. FACTUAL: A benchmark for faithful and consistent textual scene graph parsing. In ACL (Findings), pages 6377–6390. find
  16. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In ACL, pages 605–612.
  17. Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Swinbert: End-to-end transformers with sparse attention for video captioning. In CVPR, pages 17928–17937. find
  18. Zhu Liu, Teng Wang, Jinrui Zhang, Feng Zheng, Wenhao Jiang, and Ke Lu. 2023. Show, tell and rephrase: Diverse video captioning via two-stage progressive training. IEEE Trans. Multim., 25:7894–7905. find
  19. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. find
  20. Yunchuan Ma, Laiyun Qing, Guorong Li, Yuankai Qi, Amin Beheshti, Quan Z. Sheng, and Qingming Huang. 2026. RETTA: retrieval-enhanced test-time adaptation for zero-shot video captioning. Pattern Recognit., 171:112170. find
  21. David Nukrai, Ron Mokady, and Amir Globerson. 2022. Text-only training for image captioning using noise-injected CLIP. In EMNLP (Findings), pages 4055– 4063. find
  22. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318.
  24. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others.… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  25. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763.
  26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  27. Hobin Ryu, Sunghun Kang, Haeyong Kang, and Chang D. Yoo. 2021. Semantic grouping network for video captioning. In AAAI, pages 2514–2522. find
  28. Yaojie Shen, Xin Gu, Kai Xu, Heng Fan, Longyin Wen, and Libo Zhang. 2023. Accurate and fast compressed video captioning. In ICCV, pages 15558–15567. find
  29. Yoad Tewel, Yoav Shalev, Roy Nadler, Idan Schwartz, and Lior Wolf. 2023. Zero-shot video captioning by evolving pseudo-tokens. In BMVC, pages 429–432. find
  30. Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. 2022. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In CVPR, pages 17897–17907. find
  31. Mingkai Tian, Guorong Li, Yuankai Qi, Shuhui Wang, Quan Z. Sheng, and Qingming Huang. 2024. Rethink video retrieval representation for video captioning. Pattern Recognit., 156:110744. find
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008.
  33. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575.
  34. Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu. 2019a. Controllable video captioning with POS sequence guidance based on gated fusion network. In ICCV, pages 2641–2650. find
  35. Junyang Wang, Ming Yan, Yi Zhang, and Jitao Sang. 2023. From association to generation: Text-only captioning by unsupervised cross-modal mapping. In IJCAI, pages 4326–4334. find
  36. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019b. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, pages 4580–4590. find
  37. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In SIGIR, pages 641—-649.
  38. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296. find
  39. Jie Yan, Yuxiang Xie, Shiwei Zou, Yingmei Wei, and Xidao Luan. 2025. Entrocap: Zero-shot image captioning with entropy-based retrieval. Neurocomputing, 611:128666. find
  40. Bang Yang, Fenglin Liu, Xian Wu, Yaowei Wang, Xu Sun, and Yuexian Zou. 2023. Multicapclip: Autoencoding prompts for zero-shot multilingual visual captioning. In ACL, pages 11908–11922. find
  41. Hanhua Ye, Guorong Li, Yuankai Qi, Shuhui Wang, Qingming Huang, and Ming-Hsuan Yang. 2022. Hierarchical modular network for video captioning. In CVPR, pages 17918–17927. find
  42. Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Bo Chen, and Zhengjue Wang. 2024. Meacap: Memory-augmented zero-shot image captioning. In CVPR, pages 14100–14110. find
  43. Zequn Zeng, Hao Zhang, Ruiying Lu, Dongsheng Wang, Bo Chen, and Zhengjue Wang. 2023. Conzic: Controllable zero-shot image captioning by sampling-based polishing. In CVPR, pages 23465–23476. find
  44. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In ICLR. find
  45. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2025. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In CVPR, pages 9274–9285. find
  46. Qi Zheng, Chaoyue Wang, and Dacheng Tao. 2020. Syntax-aware action targeting for video captioning. In CVPR, pages 13093–13102. find
  47. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In SIGIR, pages 1097–1100.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-devil-is-distributions-explicit-modeling/