Elamur Research⌕
☰
home / papers / 2026 / MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation

MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation

Paper · 2026 · Yanghao Zhou, Haitian Li, Rexar Lin et al.

Summary

Text-to-audio-video (T2AV) systems can now produce clips in which several people hold a conversation. Existing benchmarks, however, target real recordings, general sound events or single talking heads. They miss structural errors specific to multi-party dialogue, such as a speaker's identity changing mid-clip, broken turn order, or speech assigned to the wrong person.

The authors introduce MTAVG-Bench, a benchmark built around these failure modes. A semi-automatic pipeline turns dialogue scripts into detailed prompts covering content, scene and speaker traits. Mainstream T2AV generators then render the videos, and human annotators label the errors as question–answer items. The questions are organised into a four-level taxonomy that runs from low-level signal quality up to cinematic framing.

The benchmark's main purpose is to test whether omni-modal models can detect these failures. Among 12 evaluators, Gemini 3 Pro scores best overall. Open-source models stay close on the lower-level dimensions, but all models struggle with interaction and camera-related errors. The resource supports finer comparison of generators and targeted fixes to generated video.

Key points

  • Scale: about 1,880 generated multi-talker videos and 2,410 manually annotated QA pairs.
  • Taxonomy: four levels (Signal Fidelity, Attribute Consistency, Social Interaction, Cinematic Expression) split into nine sub-dimensions:
    • VQ, SQ (video and speech quality)
    • SC, CC (scene and character consistency)
    • LS (lip synchronization)
    • SA (speaker–utterance alignment)
    • TT (turn-taking logic)
    • EA (affective and expressive alignment)
    • CA (speaker-centric camera alignment)
  • Question formats: multi-choice items with single or multiple answers for each dimension, plus head-to-head comparisons in which one video is chosen over another based on how good the dialogue is as a whole and how faithfully it follows the prompt.
  • Results: 12 proprietary and open-source omni-models were tested, and Gemini 3 Pro leads. In a turn-taking case study, Video-SALMONN2 and Qwen3-Omni missed an unscripted extra utterance, while Gemini 3 Pro detected it.
  • Stated limitations: fine-grained audio-visual synchronization errors are largely out-of-distribution for current models. Failure types are also unevenly distributed across generators, which complicates SFT or RL training of evaluators.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksDialogue systems · 624 worksSpeech & spoken language · 480 works

Cite

Cite as Yanghao Zhou, Haitian Li, Rexar Lin, Heyan Huang, Jinxing Zhou, Changsen Yuan, Tian Lan, Ziqin Zhou, Yudong Li, Jiajun Xu, Jingyun Liao, YiMing Cheng, Xuefeng Chen, Xian-Ling Mao and Yousheng Feng. 2026. MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.913

@inproceedings{zhou2026mtavg,
  title = {{MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation}},
  author = {Yanghao Zhou, Haitian Li, Rexar Lin, Heyan Huang, Jinxing Zhou, Changsen Yuan, Tian Lan, Ziqin Zhou, Yudong Li, Jiajun Xu, Jingyun Liao, YiMing Cheng, Xuefeng Chen, Xian-Ling Mao and Yousheng Feng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.913},
  url = {https://doi.org/10.18653/v1/2026.acl-long.913}
}

References

45 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Inclusion AI, Bowen Ma, Cheng Zou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Dandan Zheng, Fudong Wang, Furong Xu, and 1 others. 2025. Ming-flash-omni: A sparse, unified architecture for multimodal perception and generation. arXiv preprint arXiv:2510.24821. — [arXiv:2510.24821](https://arxiv.org/abs/2510.24821) find
  2. Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu. 2025. Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters. arXiv preprint arXiv:2505.20156. — [arXiv:2505.20156](https://arxiv.org/abs/2505.20156) find
  3. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and 1 others. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476. find
  4. Sanjoy Chowdhury, Karren D Yang, Xudong Liu, Fartash Faghri, Pavan Kumar Anasosalu Vasu, Oncel Tuzel, Dinesh Manocha, Chun-Liang Li, and Raviteja Vemulapalli. 2025. Amuse: Audio-visual benchmark and alignment framework for agentic multi-speaker understanding. arXiv preprint arXiv:2512.16250. — [arXiv:2512.16250](https://arxiv.org/abs/2512.16250) find
  5. Yikang Ding, Jiwen Liu, Wenyuan Zhang, Zekun Wang, Wentao Hu, Liyuan Cui, Mingming Lao, Yingchao Shao, Hui Liu, Xiaohan Li, and 1 others. 2025. Kling-avatar: Grounding multimodal instructions for cascaded long-duration avatar animation synthesis. arXiv preprint arXiv:2509.09595. — [arXiv:2509.09595](https://arxiv.org/abs/2509.09595) find
  6. Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, and Steven Hoi. 2025. Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation. arXiv preprint arXiv:2506.18866. — [arXiv:2506.18866](https://arxiv.org/abs/2506.18866) find
  7. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190.
  8. Google. 2025. Gemini: a family of highly capable multimodal models. https://blog.google/products/gemini/gemini-3-flash/. ↗
  9. Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, and 1 others. 2026. Ltx-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233. — [arXiv:2601.03233](https://arxiv.org/abs/2601.03233) find
  10. Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, and Ran Yi. 2025. Harmony: Harmonizing audio and video generation through cross-task synergy. arXiv preprint arXiv:2511.21579. — [arXiv:2511.21579](https://arxiv.org/abs/2511.21579) find
  11. Daili Hua, Xizhi Wang, Bohan Zeng, Xinyi Huang, Hao Liang, Junbo Niu, Xinlong Chen, Quanqing Xu, and Wentao Zhang. 2025. Vabench: A comprehensive benchmark for audio-video generation. arXiv preprint arXiv:2512.09299. — [arXiv:2512.09299](https://arxiv.org/abs/2512.09299) find
  12. Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. 2025. Let them talk: Audio-driven multi-person conversational video generation. arXiv preprint arXiv:2505.22647. — [arXiv:2505.22647](https://arxiv.org/abs/2505.22647) find
  13. Tian Lan, Yang-Hao Zhou, Zi-Ao Ma, Fanshu Sun, Rui-Qing Sun, Junyu Luo, Rong-Cheng Tu, Heyan Huang, Chen Xu, Zhijing Wu, and 1 others. 2025. A survey of automatic evaluation methods on text, visual and speech generations. arXiv preprint arXiv:2506.10019. — [arXiv:2506.10019](https://arxiv.org/abs/2506.10019) find
  14. Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, and Tao Huang. 2025a. Pistachio: Towards synthetic, balanced, and long-form video anomaly benchmarks. arXiv preprint arXiv:2511.19474. — [arXiv:2511.19474](https://arxiv.org/abs/2511.19474) find
  15. Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Fuxiao Liu, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. 2025b. Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding. arXiv preprint arXiv:2505.01481. — [arXiv:2505.01481](https://arxiv.org/abs/2505.01481) find
  16. Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, and 1 others. Javisgpt: A unified multi-modal llm for sounding-video comprehension and generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Kai… — [arXiv:2503.23377](https://arxiv.org/abs/2503.23377) find
  17. Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. 2025b. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328. — [arXiv:2502.04328](https://arxiv.org/abs/2502.04328) find
  18. Chetwin Low, Weimin Wang, and Calder Katyal. 2025. Ovi: Twin backbone cross-modal fusion for audio-video generation. arXiv preprint arXiv:2510.01284. — [arXiv:2510.01284](https://arxiv.org/abs/2510.01284) find
  19. Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. 2024. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6607–6616. find
  20. Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, Se-ungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, and 1 others. 2025. See, hear, and understand: Benchmarking audiovisual human speech understanding in multimodal large language models. arXiv preprint… — [arXiv:2512.02231](https://arxiv.org/abs/2512.02231) find
  21. Federico Nocentini, Thomas Besnier, Claudio Ferrari, Sylvain Arguillere, Mohamed Daoudi, and Stefano Berretti. 2024. Beyond fixed topologies: Unregistered training and comprehensive evaluation metrics for 3d talking heads. arXiv preprint arXiv:2410.11041. on Computer Vision and Pattern Recognition,… — [arXiv:2410.11041](https://arxiv.org/abs/2410.11041) find
  22. Shuai Wang, Zhaokai Sun, Zhennan Lin, Chengyou Wang, Zhou Pan, and Lei Xie. 2025c. Msu-bench: Towards understanding the conversational multi-talker scenarios. arXiv preprint arXiv:2508.08155. — [arXiv:2508.08155](https://arxiv.org/abs/2508.08155) find
  23. OpenAI. 2025. Sora 2: Video generation model. find
  24. Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, and 1 others. 2025. Mocha: Towards movie-grade talking character synthesis. arXiv preprint arXiv:2503.23307. — [arXiv:2503.23307](https://arxiv.org/abs/2503.23307) find
  25. KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492. find
  26. Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. 2025. Video models are zero-shot learners and reasoners. CoRR, abs/2509.20328. — [arXiv:2509.20328](https://arxiv.org/abs/2509.20328) find
  27. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 5370–5381. find
  28. Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. 2025. video-salmonn 2: Captioning-enhanced audio-visual large language models. arXiv preprint arXiv:2506.15220. — [arXiv:2506.15220](https://arxiv.org/abs/2506.15220) find
  29. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025a. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  30. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. Preprint, arXiv:2509.17765.
  31. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  32. Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. 2025. Audio-centric video understanding benchmark without text shortcut. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6580–6…
  33. OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, and 1 others. 2026. Mova: Towards scalable and synchronized video-audio generation. arXiv preprint arXiv:2602.08794. — [arXiv:2602.08794](https://arxiv.org/abs/2602.08794) find
  34. Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, and 1 others. 2025. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. arXiv preprint arXiv:2509.18154.
  35. Alibaba Tongyi. 2025. Wan 2.5: Unified multi-modal video generation framework. find
  36. Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, and Limin Wang. 2025. Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions. arXiv preprint arXiv:2511.03334. — [arXiv:2511.03334](https://arxiv.org/abs/2511.03334) find
  37. Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, and Daxin Jiang. 2025a. 1: Unified audio-video generation via stitching of experts. arXiv preprint arXiv:2509.06155, 1(2):3. — [arXiv:2509.06155](https://arxiv.org/abs/2509.06155) find
  38. Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. 2025b. Universe-1: Unified audio-video generation via stitching of experts. arXiv preprint arXiv:2509.06155. — [arXiv:2509.06155](https://arxiv.org/abs/2509.06155) find
  39. Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. 2023. Seeing what you said: Talking face generation guided by a lip reading expert. In Proceedings of the IEEE/CVF Conference Weixia Zhang, Chengguang Zhu, Jingnan Gao, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. 2024. A comparat… find
  40. Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186. find
  41. Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, and Tiejun Zhao. 2025. Interactive conversational head generation. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  42. Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. 2022. Responsive listening head generation: a benchmark dataset and baseline. In European conference on computer vision, pages 124– 142. Springer. find
  43. Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. 2023. Learning and evaluating human preferences for conversational head generation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 9615–9619. find
  44. Yingjie Zhou, Zicheng Zhang, Wei Sun, Xiaohong Liu, Xiongkuo Min, Zhihua Wang, Xiao-Ping Zhang, and Guangtao Zhai. 2024. Thqa: A perceptual quality assessment database for talking heads. In 2024 IEEE International Conference on Image Processing (ICIP), pages 15–21. IEEE. find
  45. Zeyu Zhu, Weijia Wu, and Mike Zheng Shou. 2025. Multi-human interactive talking dataset. arXiv preprint arXiv:2508.03050. — [arXiv:2508.03050](https://arxiv.org/abs/2508.03050) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-mtavg-bench-diagnostic-benchmark-multi/