Elamur Research⌕
☰
home / papers / 2026 / From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)

From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)

Paper · 2026 · Suyash Mishra, Qiang Li, Anubhav Girdhar et al.

Summary

Pharmaceutical companies produce many long videos, such as clinical trial interviews, seminars, manufacturing workflows and conference recordings. Reviewing and repurposing them by hand takes a lot of time. Generative video tools are a poor fit because this domain requires original frames, original audio, exact source timestamps and auditability. The authors build an extractive Video-to-Video clip generation framework that combines Audio-Language Models (Whisper transcription) with VLMs and LLMs (Claude 3, Gemini, GPT-4o, Qwen) to pick and stitch highlight segments. It has three parts:

  • a Cut & Merge algorithm that normalizes timestamps and adds fade-in/out to avoid jump cuts and audio glitches;
  • role definitions and prompt injection to tailor clips for marketing, training or regulatory audiences;
  • a cost-aware pipeline that relies mainly on transcripts and calls VLMs only for visually heavy content.

The framework was evaluated on Video-MME and a proprietary pharma corpus. It shows roughly 3–4× faster generation and about 4× lower cost than strong VLM baselines such as Gemini 2.5 Pro, with competitive clip quality. This makes highlight generation for regulated life-science content practical and traceable.

Key points

  • Data: Video-MME (900 videos) plus a proprietary set of 16,159 pharmacy videos.
    • The proprietary set covers 14 disease areas, more than 20 languages and 8 video formats.
    • Durations run from under 2 minutes to over 3 hours.
  • Models compared: more than 11 VLMs, including the Gemini 1.5/2.0/2.5 family, Qwen-7B-VL and Qwen-72B, Claude 3.5 Sonnet and GPT-4o. Whisper-large V2 and Whisper-turbo V3 handle transcription, with V2 acting as a safeguard.
  • Efficiency: the method needs about 30–55 s per video, compared with about 120 s for one baseline (apparently Gemini 2.5 Pro, judging by its token pricing). Token pricing is $0.3/$2.5 versus $1.25/$10 per million input/output tokens.
  • Quality:
    • The proposed method has the highest informativeness (0.721) and the lowest redundancy (0.339).
    • It reaches a coherence score of 0.348. In the reported table, the baselines score higher on coherence (0.446 and 0.410), even though the abstract says the method improved coherence. This discrepancy should be checked against the full paper.
  • Findings and release:
    • A transcript-only modality already gives fast, cheap, good clips.
    • Gemini models tend to produce fragmented segments.
    • Rich prompt injection (agenda, keywords, pauses, transitions) beats naive concatenation.
    • Role prompts are released in the appendix, and the Cut & Merge algorithm is patent-pending (EPO 25175653.2).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 worksPharmacology, Toxicology and Pharmaceutics · 7 worksPharmacology · 1 work

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Suyash Mishra, Qiang Li, Anubhav Girdhar and Srikanth Patil. 2026. From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs). Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.customnlp4u-1.3

@inproceedings{mishra2026understanding,
  title = {{From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)}},
  author = {Suyash Mishra, Qiang Li, Anubhav Girdhar and Srikanth Patil},
  booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)},
  year = {2026},
  doi = {10.18653/v1/2026.customnlp4u-1.3},
  url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.3}
}

References

41 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Toqa Alaa, Ahmad Mongy, Assem Bakr, Mariam Diab, and Walid Gomaa. 2024. Video summarization techniques: A comprehensive review. Preprint, arXiv:2410.04449. — [arXiv:2410.04449](https://arxiv.org/abs/2410.04449) find
  2. Yawen Guo, Xiao Liu, Anjana Susarla, and Rema Padman. 2024a. Go to youtube and call me in the morning: Use of social media for chronic conditions. Management Information Systems Quarterly. find
  3. Animoto. 2024. Animoto: Image + text → video clip generation. Web-based video creator; builds short visuals using static images and prompts/text. find
  4. Yawen Guo, Xiao Liu, Anjana Susarla, and Rema Padman. 2024b. Go to youtube and call me in the morning: Use of social media for chronic conditions. Management Information Systems Quarterly. find
  5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  6. HeyGen. 2024. Heygen: Ai avatar-based video clip generation. find
  7. Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. 2018. Recycle-gan: Unsupervised video Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2022. Video diffusion models. In NeurIPS. find
  8. Ming Hu, Kun Yuan, Yaling Shen, Feilong Tang, Xiaohao Xu, Lin Zhou, Wei Li, Ying Chen, Zhongxing Xu, Zelin Peng, Siyuan Yan, Vinkle Srivastav, Diping Song, Tianbin Li, Danli Shi, Jin Ye, Nicolas Padoy, Nassir Navab, Junjun He, and Zongyuan Ge. 2024. Ophclip: Hierarchical retrieval-augmented learning… — [arXiv:2411.15421](https://arxiv.org/abs/2411.15421) find
  9. Corin Otesteanu, Martina Ugrinic, Gregor Holzner, Yun-Tsan Chang, Christina Fassnacht, Emmanuella Guen-ova, Stavros Stavrakis, Andrew deMello, and Manfred Claassen. 2021. A weakly supervised deep learning approach for label-free imaging flow-cytometry-based blood diagnostics. Cell Reports Methods. find
  10. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE. find
  11. Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. 2024. V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning. arXiv preprint arXiv:2404.12353. — [arXiv:2404.12353](https://arxiv.org/abs/2404.12353) find
  12. Songtao Jiang, Yuan Wang, Sibo Song, Yan Zhang, Zijie Meng, Bohan Lei, Jian Wu, Jimeng Sun, and Zuozhu Liu. 2025. Omniv-med: Scaling medical vision-language model for universal visual understanding. arXiv preprint arXiv:2504.14692. find
  13. Pika Labs. 2024. Pika labs: Animate still images or frames into video. find
  14. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision (whisper).
  15. Kaiber AI. 2024. Kaiber ai: Single image-to-video with prompt-driven animation. find
  16. RunwayML. 2023. Runway gen-2. https://runwayml.com/. Accessed: 2025-06-01. ↗
  17. Min Jung Lee, Dayoung Gong, and Minsu Cho. 2025a. Video summarization with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  18. Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. 2024. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485. find
  19. Min Jung Lee, Dayoung Gong, and Minsu Cho. 2025b. Video summarization with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  20. Murali Srinivasan, Porawit Kamnoedboon, Dusit Nan-tanapiboon, Piero Papi, and Umberto Romeo. 2025. Non-surgical management of peri-implantitis with photodynamic therapy: A systematic review and meta-analysis of clinical parameters and biomarkers. Journal of Dentistry. find
  21. Zhuoling Li, Hossein Rahmani, Qiuhong Ke, and Jun Liu. 2025. Longdiff: Training-free long video generation in one go. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  22. Stability AI. 2024. Stable video diffusion: Image-to-video generation from a single frame. find
  23. Kangning Liu, Shuhang Gu, Andres Romero, and Radu Timofte. 2021. Unsupervised multimodal video-to-video translation via self-supervised learning. In Proceedings of the IEEE Conference on Applications of Computer Vision (WACV). find
  24. Synthesia. 2024. Synthesia: Ai avatars and voice-based video clips. find
  25. Amara Tariq, Rimita Lahiri, Charles Kahn, and Imon Banerjee. 2025. Position: Restructuring of categories and implementation of guidelines essential for vlm adoption in healthcare. In Proceedings of the 2025 Conference on Vision-Language Models in Healthcare. find
  26. Tianrui Liu, Qingjie Meng, Athanasios Vlontzos, Jeremy Tan, Daniel Rueckert, and Bernhard Kainz. 2020. Ultrasound video summarization using deep reinforcement learning. arXiv preprint arXiv:2005.09531. — [arXiv:2005.09531](https://arxiv.org/abs/2005.09531) find
  27. Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2016. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (NeurIPS), volume 29. find
  28. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281.
  29. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  30. TingChun Wang, MingYu Liu, JunYan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 31. find
  31. CoRR, OpenAI. 2023. Introducing whisper large-v3. OpenAI Blog / Model Card. find
  32. Yimu Wang, Xuye Liu, Wei Pang, Li Ma, Shuai Yuan, Paul Debevec, and Ning Yu. 2025. Survey of video diffusion models: Foundations, implementations, and applications. arXiv preprint arXiv:2504.16081. — [arXiv:2504.16081](https://arxiv.org/abs/2504.16081) find
  33. OpusClip. 2023. Opusclip: Ai-powered clip generation from long-form video content. find
  34. Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. 2024. Longvlm: Efficient long video understanding via large language models. arXiv preprint arXiv:2404.03384. — [arXiv:2404.03384](https://arxiv.org/abs/2404.03384) find
  35. Siqi Wu, Marian-Andrei Rizoiu, and Lexing Xie. 2018. Beyond views: Measuring and predicting engagement in online videos. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). find
  36. Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. 2023. A survey on video diffusion models. ACM Computing Surveys. find
  37. Xu Yan, Xinyang Zhang, Pieter Abbeel, and Aravind Srinivas. 2021. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157. — [arXiv:2104.10157](https://arxiv.org/abs/2104.10157) find
  38. Shentao Yang, Haichuan Yang, Linna Du, Adithya Ganesh, Bo Peng, Boying Liu, Serena Li, and Ji Liu. 2025a. Swat: Statistical modeling of video watch time through user behavior analysis. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25. Association f… find
  39. Xindi Yang, Baolu Li, Yiming Zhang, Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang, Jianfei Cai, Tien-Tsin Wong, Huchuan Lu, and Xu Jia. 2025b. Towards physically plausible video generation via vlm planning. arXiv preprint arXiv:2503.23368. — [arXiv:2503.23368](https://arxiv.org/abs/2503.23368) find
  40. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. find
  41. Zachary Zulko. 2015. Moviepy: Video editing with python. https://github.com/Zulko/moviepy. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2026-understanding-engagement-personalized-pharmacy-video/