Elamur Research⌕
☰
home / papers / 2026 / SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation

SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation

Paper · 2026 · Chengyi Yang, Pengzhen Li, Jiayin Qi et al.

Summary

Text-to-video (T2V) diffusion models often fail on demanding prompts, such as abstract ideas, intricate spatial layouts, crowded scenes, physical causality or long temporal processes. Short, underspecified user prompts make these failures worse. Earlier prompt-rewriting methods were mostly built for images, or they expand text through retrieval based on surface similarity, without identifying what makes a prompt hard.

The authors propose SCMAPR, a training-free, model-agnostic pipeline in which six LLM-driven agents work through five stages:

  1. A router assigns each prompt to one of ten complex-scenario categories or to a non-difficult tag.
  2. A generator drafts a rewriting policy for that specific prompt.
  3. A refiner rewrites the prompt according to the policy.
  4. A verification group decomposes the user's intent into semantic atoms and checks entailment.
  5. The rewrite is revised only when fidelity violations are found.

The authors also introduce T2V-Complexity, a benchmark containing only complex-scenario prompts, balanced across the ten categories. On VBench, EvalCrafter, T2V-CompBench and T2V-Complexity, SCMAPR outperforms three state-of-the-art baselines. The results suggest that diagnosing difficulty before rewriting improves both text-video alignment and visual quality.

Key points

  • Ten categories make up the taxonomy: Camera Motion, Complex Spatial Relations, Temporal Consistency, Abstract Descriptions, Object Interaction, Fine-Grained Appearance, Scene Transitions, Multi-Element Scenes, Causality & Physics and Stylistic Hybrids.
  • Reported gains over the baselines reach 2.67% in average VBench score, 3.28 on EvalCrafter and 0.028 on T2V-CompBench; LaVie and Wan are among the video models tested.
  • On T2V-Complexity, the average VBench-metric score rises from 82.95% to 85.69% compared with direct prompting, with improvements on all six metrics.
  • In ablations on VBench, removing scenario routing costs 1.72%, removing verification and self-correction costs 0.58%, and removing policy generation costs 0.46%.
  • Stated limitations are extra inference cost and dependence on the LLM's reasoning ability; the code is available at github.com/HiThink-Research/SCMAPR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksComputer Graphics and Computer-Aided Design · 16 works

Cite

Cite as Chengyi Yang, Pengzhen Li, Jiayin Qi, Aimin Zhou, Ji Wu and Ji Liu. 2026. SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.345

@inproceedings{yang2026scmapr,
  title = {{SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation}},
  author = {Chengyi Yang, Pengzhen Li, Jiayin Qi, Aimin Zhou, Ji Wu and Ji Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.345},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.345}
}

References

46 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Conf. on Neural Information Processing Systems (NeurIPS). find
  2. Stephen Brade, Bryan Wang, Maurício Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Annual ACM Symposium on User Interface Software and Technology (UIST), pages 96:1–96:14. find
  3. Kai He, Chin-Hsuan Wu, and Igor Gilitschenski. 2025. CTRL-D: controllable dynamic 3d scene editing with personalized 2d diffusion. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 26630–26640. find
  4. Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Bytheway: Boost your text-to-video generation model to higher quality in a training-free way. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 12999–13008. find
  5. Chi-Pin Huang, Yen-Siang Wu, Hung-Kai Chung, Kai-Po Chang, Fu-En Yang, and Yu-Chiang Frank Wang. 2025a. Videomage: Multi-subject and motion customization of text-to-video diffusion models. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 17603–17612. find
  6. Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. 2024a. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 7310–7320. find
  7. Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. 2025b. MIDI: multi-instance diffusion for single image to 3d scene generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 23646– 23657. find
  8. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics (ACL Findings), pages 2318–2335…
  9. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, and 263 other authors. 2025. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  10. Tiehan Fan, Kepan Nan, Rui Xie, Penghao Zhou, Zhenheng Yang, Chaoyou Fu, Xiang Li, Jian Yang, and Ying Tai. 2025. Instancecap: Improving text-to-video generation via instance-aware structured caption. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 28974–28983. find
  11. Weixi Feng, Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, and Weili Nie. 2025. Blobgen-vid: Compositional text-to-video generation with blob video representations. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 12989–12998. find
  12. Bingjie Gao, Xinyu Gao, Xiaoxue Wu, Yujie Zhou, Yu Qiao, Li Niu, Xinyuan Chen, and Yaohui Wang. 2025. The devil is in the prompts: Retrieval-augmented prompt optimization for text-to-video generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3173–3183. find
  13. Hao Guo, Xiaoshui Huang, Jiacheng Hao, Yunpeng Bai, Hongping Gan, and Yilei Shi. 2025. Brepgiff: Lightweight generation of complex b-rep with 3d GAT diffusion. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 26587– 26596. find
  14. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2023. Optimizing prompts for text-to-image generation. In Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and… find
  15. Diljeet Jagpal, Xi Chen, and Vinay P. Namboodiri. 2025. EIDT-V: exploiting intersections in diffusion trajectories for model-agnostic, zero-shot, training-free text-to-video generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 18219–18228. find
  16. Che Hyun Lee, Heeseung Kim, Jiheum Yeom, and Sungroh Yoon. 2025. Editext: Controllable coarse-to-fine text editing with diffusion language models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 22798–22815. find
  17. Mingcheng Li, Xiaolu Hou, Ziyang Liu, Dingkang Yang, Ziyun Qian, Jiawei Chen, Jinjie Wei, Yue Jiang, Qingyao Xu, and Lihua Zhang. 2025. MCCD: Multi-agent collaboration-based compositional diffusion for complex text-to-image generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CV… find
  18. Jiantao Lin, Xin Yang, Meixi Chen, Yingjie Xu, Dongyu Yan, Leyi Wu, Xinli Xu, Lie Xu, Shunsi Zhang, and Ying-Cong Chen. 2025. Kiss3dgen: Repurposing image diffusion models for 3d asset generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 5870–5880. find
  19. Che Liu, Yingji Zhang, Dong Zhang, Weijie Zhang, Chenggong Gong, Yu Lu, Shilin Zhou, Ziliang Gan, Ziao Wang, Haipang Wu, and 1 others. 2025. Nexus-o: An omni-perceptive and-interactive model for language, audio, and vision. In ACM Int. Conf. on Multimedia (MM), pages 10787–10796. 6951 10 find
  20. Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, and Dejing Dou. 2024a. Fisher information-based efficient curriculum federated learning with large language models. In Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 10497–10523. find
  21. Weizhi Liu, Yue Li, Dongdong Lin, Hui Tian, and Haizhou Li. 2024b. GROOT: generating robust watermark for diffusion-model-based audio synthesis. In ACM Int. Conf. on Multimedia (MM), pages 3294– 3302. find
  22. Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. 2024c. EvalCrafter: benchmarking and evaluating large video generation models. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 22139–22149. find
  23. Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. 2023. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In Conf. on Neural Information Processing Systems (NeurIPS). find
  24. Hyung-Seok Oh, Sang-Hoon Lee, and Seong-Whan Lee. 2024. Diffprosody: Diffusion-based latent prosody generation for expressive speech synthesis with prosody conditional adversarial training. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2654–2666. find
  25. Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Jingun Kwon, Hidetaka Kamigaito, Katsuhiko Hayashi, Manabu Okumura, and Taro Watanabe. 2025. Text-tiger: Text-based intelligent generation with entity prompt refinement for text-to-image generation. arXiv preprint arXiv:2504.18269. — [arXiv:2504.18269](https://arxiv.org/abs/2504.18269) find
  26. Guy Parsons. 2022. The dall·e 2 prompt book. https://dallery.gallery/the-dalle2-prompt-book. ↗
  27. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In IEEE/CVF Int. Conf. on Computer Vision (ICCV), pages 4195–4205.
  28. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695.
  29. Yifan Sun, Jean-Baptiste Tien, and 1 others. 2024. Retrieval augmented prompt optimization. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. find
  30. Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, and 42 others. 2025a. Wan: Open and advanced large-scale video gene…
  31. Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, Peter Vajda, Niraj K. Jha, and Xiaoliang Dai. 2025b. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity. In IEEE/C… find
  32. Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. 2025c. LaVie: High-quality video generation with casca… find
  33. Yuanzhi Wang, Yong Li, Mengyi Liu, Xiaoya Zhang, Xin Liu, Zhen Cui, and Antoni B. Chan. 2025d. Re-attentional controllable video diffusion editing. In AAAI Conf. on Artificial Intelligence (AAAI), pages 8123–8131. find
  34. Sam Witteveen and Martin Andrews. 2022. Investigating prompt engineering in diffusion models. arXiv preprint arXiv:2211.15462. — [arXiv:2211.15462](https://arxiv.org/abs/2211.15462) find
  35. Qiyao Xue, Xiangyu Yin, Boyuan Yang, and Wei Gao. 2025. Phyt2v: Llm-guided iterative self-refinement for physics-grounded text-to-video generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 18826–18836. find
  36. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 2023. Make-a-video: Text-to-video generation without text-video data. In Int. Conf. on Learning Representations (ICLR), pages 1–16. find
  37. Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. 2025. CogVideoX: Text-to-video diffusion models with an expert transformer.…
  38. Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. 2025. T2V-CompBench: A comprehensive benchmark for compositional text-to-video generation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 8406–8416. find
  39. Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyang Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. 2025. Identity-preserving text-to-video generation by frequency decomposition. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 12978–12988. 6952 11 find
  40. Jingtao Zhan, Qingyao Ai, Yiqun Liu, Jia Chen, and Shaoping Ma. 2024a. Capability-aware prompt reformulation learning for text-to-image generation. In Int. ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 2145–2155. find
  41. Jingtao Zhan, Qingyao Ai, Yiqun Liu, Yingwei Pan, Ting Yao, Jiaxin Mao, Shaoping Ma, and Tao Mei. 2024b. Prompt refinement with image pivot for text-to-image generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 941–954. find
  42. Shengjun Zhang, Jinzhao Li, Xin Fei, Hao Liu, and Yueqi Duan. 2025a. Scene splatter: Momentum 3d scene generation from single image with video diffusion model. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 6089–6098. find
  43. Yabo Zhang, Yuxiang Wei, Xianhui Lin, Zheng Hui, Peiran Ren, Xuansong Xie, and Wangmeng Zuo. 2025b. Videoelevator: Elevating video generation quality with versatile text-to-image diffusion models. In AAAI Conf. on Artificial Intelligence (AAAI), pages 10266–10274. find
  44. Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. 2024. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404. — [arXiv:2412.20404](https://arxiv.org/abs/2412.20404) find
  45. Wenqiao Zhu, Ji Liu, Lulu Wang, Jun Wu, and Yulun Zhang. 2025a. Sgdpo: Self-guided direct preference optimization for language model alignment. In Findings of the Association for Computational Linguistics (ACL Findings), pages 12366–12383. find
  46. Wenqiao Zhu, Ji Liu, Rongjunchen Zhang, Haipang Wu, and Yulun Zhang. 2025b. CARFT: boosting llm reasoning via contrastive learning with annotated chain-of-thought-based reinforced fine-tuning. In Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 5933–5948. 6953 12 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-scmapr-self-correcting-multi-agent/