Elamur Research⌕
☰
home / papers / 2023 / NExT-GPT: Any-to-Any Multimodal LLM

NExT-GPT: Any-to-Any Multimodal LLM

Paper · 2023 · Shengqiong Wu, Hao Fei, Leigang Qu et al.

Summary

Most multimodal LLMs can take in images, video or audio but can only answer in text. NExT-GPT (Wu et al., NUS, ICML 2024) aims to accept and produce any mix of text, image, video and audio within one end-to-end trained system. Its design has three tiers:

  • Input: pre-trained encoders such as ImageBind turn inputs into features, and projection layers map those features into the LLM's space.
  • Core: an open-source LLM reasons over the input and emits text plus special "modality signal" tokens.
  • Output: these tokens are projected to condition off-the-shelf diffusion decoders for each output modality.

The encoders, decoders and most of the LLM stay frozen, so only about 1% of parameters are trained, mainly the projection layers. Training uses two lightweight alignment steps, one at the encoding side and one at the decoding side. The authors then apply modality-switching instruction tuning (MosIT) with LoRA on a manually annotated dataset. In human evaluation the model beat tool-chaining pipeline systems at following complex instructions. The work matters because it shows that unified any-to-any assistants can be built cheaply and extended to new modalities.

Key points

  • The MosIT dataset has 5,000 hand-curated multi-turn samples covering varied modality combinations. Fine-tuning with LoRA updates the input and output projections plus some LLM weights.
  • With grouping-based input projection, NExT-GPT scores 66.7 on VQA v2, 48.4 on VizWiz, 64.5 on MSVD-QA, 61.4 on MSRVTT-QA and 81.3 on AudioCaps. A plain linear layer and Q-former + linear projections both score lower.
  • On signal-token ablations, images and audio need only 4 and 8 tokens respectively for satisfactory generation, while video benefits from more.
  • For human evaluation, 100 complex instructions were generated with GPT-4 and five volunteers rated the outputs. Raters judged instruction following, rationality and image quality, comparing NExT-GPT with two pipeline systems and a caption-only variant (NExT-GPT-caption).
  • The authors warn of possible hallucinated output, and the released model prohibits commercial use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua. 2023. NExT-GPT: Any-to-Any Multimodal LLM. arXiv:2309.05519.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.05519 · arXiv 2309.05519

@misc{wu2023next,
  title = {{NExT-GPT: Any-to-Any Multimodal LLM}},
  author = {Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji and Tat-Seng Chua},
  year = {2023},
  doi = {10.48550/arxiv.2309.05519},
  eprint = {2309.05519},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.05519}
}

References

127 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, H., Anderson, P., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., and Lee, S. nocaps: novel object captioning at scale. In Proceedings of the ICCV, pp. 8947–8956, 2019. find
  2. Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… — Flamingo: a Visual Language Model for Few-Shot Learning
  3. An, J., Zhang, S., Yang, H., Gupta, S., Huang, J., Luo, J., and Yin, X. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. CoRR, abs/2304.08477, 2023. — [arXiv:2304.08477](https://arxiv.org/abs/2304.08477) find
  4. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the CVPR, pp. 6077–6086, 2018. find
  5. Avrahami, O., Fried, O., and Lischinski, D. Blended latent diffusion. ACM Trans. Graph., 42(4):149:1–149:11, 2023. find
  6. Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the ICCV, pp. 1708–1718, 2021. find
  7. Bashiri, M., Walker, E. Y., Lurz, K., Jagadish, A., Muhammad, T., Ding, Z., Ding, Z., Tolias, A. S., and Sinz, F. H. A flow-based latent state generative model of neural population responses to natural images. In Proceedings of the NeurIPS, pp. 15801–15815, 2021. find
  8. Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In Proceedings of the ICLR, 2019. find
  9. Cerspense. Zeroscope: Diffusion-based text-to-video synthesis. 2023. URL https://huggingface.co/cerspense. ↗
  10. Ceylan, D., Huang, C. P., and Mitra, N. J. Pix2video: Video editing using image diffusion. CoRR, abs/2303.12688, 2023. — [arXiv:2303.12688](https://arxiv.org/abs/2303.12688) find
  11. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90 2023. find
  12. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J.,… find
  13. Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. Diffedit: Diffusion-based semantic image editing with mask guidance. In Proceedings of the ICLR, 2023. find
  14. Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. C. H. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, abs/2305.06500, 2023.
  15. Dessı̀, R., Bevilacqua, M., Gualdoni, E., Rakotonirina, N. C., Franzon, F., and Baroni, M. Cross-domain image captioning with discriminative finetuning. In Proceedings of the CVPR, pp. 6935–6944, 2023. find
  16. Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., and Tang, J. Cogview: Mastering text-to-image generation via transformers. In Proceedings of the NeurIPS, pp. 19822–19835, 2021.
  17. Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., Kong, X., Zhang, X., Ma, K., and Yi, L. Dreamllm: Synergistic multimodal comprehension and creation. CoRR, abs/2309.11499, 2023. — [arXiv:2309.11499](https://arxiv.org/abs/2309.11499) find
  18. Dorkenwald, M., Milbich, T., Blattmann, A., Rombach, R., Derpanis, K. G., and Ommer, B. Stochastic image-to-video synthesis using cinns. In Proceedings of the CVPR, pp. 3742–3753, 2021. find
  19. Fan, W., Chen, Y., Chen, D., Cheng, Y., Yuan, L., and Wang, Y. F. Frido: Feature pyramid diffusion for complex scene image synthesis. CoRR, abs/2208.13753, 2022. — [arXiv:2208.13753](https://arxiv.org/abs/2208.13753) find
  20. Feng, W., He, X., Fu, T., Jampani, V., Akula, A. R., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y. Training-free structured diffusion guidance for compositional text-to-image synthesis. CoRR, abs/2212.05032, 2022. — [arXiv:2212.05032](https://arxiv.org/abs/2212.05032) find
  21. Feng, W., Zhu, W., Fu, T., Jampani, V., Akula, A. R., He, X., Basu, S., Wang, X. E., and Wang, W. Y. Layoutgpt: Compositional visual planning and generation with large language models. CoRR, abs/2305.15393, 2023. — [arXiv:2305.15393](https://arxiv.org/abs/2305.15393) find
  22. Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. CoRR, abs/2208.01618, 2022. find
  23. Ge, S., Hayes, T., Yang, H., Yin, X., Pang, G., Jacobs, D., Huang, J., and Parikh, D. Long video generation with time-agnostic VQGAN and time-sensitive transformer. In Proceedings of the ECCV, pp. 102–118, 2022. find
  24. Ge, Y., Ge, Y., Zeng, Z., Wang, X., and Shan, Y. Planting a SEED of vision in large language model. CoRR, abs/2307.08041, 2023. find
  25. Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In Proceedings of the ICASSP, pp. 776–780, 2017. find
  26. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. CoRR, abs/2305.05665, 2023.
  27. Gontier, F., Serizel, R., and Cerisara, C. Automated audio captioning by fine-tuning BART with audioset tags. In Proceedings of the DCASE, pp. 170–174, 2021. find
  28. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the CVPR, pp. 6325–6334, 2017.
  29. Gu, X., Chen, G., Wang, Y., Zhang, L., Luo, T., and Wen, L. Text with knowledge graph augmented transformer for video captioning. In Proceedings of the CVPR, pp. 18941–18951, 2023. find
  30. Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the CVPR, pp. 3608–3617, 2018. find
  31. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prompt image editing with cross-attention control. In Proceedings of the ICLR, 2023. find
  32. Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. CoRR, abs/2205.15868, 2022.
  33. Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, 2021. find
  34. Hsu, W., Bolte, B., Tsai, Y. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29:3451–3460, 2021.
  35. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In Proceedings of the ICLR, 2022.
  36. Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In Proceedings of the ICML, pp. 13916–13932, 2023a. find
  37. Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., Ren, Y., Zhao, Z., and Watanabe, S. Audiogpt: Understanding and generating speech, music, sound, and talking head. CoRR, abs/2304.12995, 2023b. — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) find
  38. Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., and Wei, F. Language is not all you need: Aligning perception with language models. CoRR, abs/2302.14045, 2023c.
  39. Huang, W., Tu, S., and Xu, L. Pfb-diff: Progressive feature blending diffusion for text-driven image editing. CoRR, abs/2306.16894, 2023d. — [arXiv:2306.16894](https://arxiv.org/abs/2306.16894) find
  40. Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):664–676, 2017. find
  41. Karras, J., Holynski, A., Wang, T., and Kemelmacher-Shlizerman, I. Dreampose: Fashion image-to-video synthesis via stable diffusion. CoRR, abs/2304.06025, 2023. — [arXiv:2304.06025](https://arxiv.org/abs/2304.06025) find
  42. Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Generating captions for audios in the wild. In Proceedings of the NAACL, pp. 119–132, 2019.
  43. Kim, E., Kim, J., Oh, Y., Kim, K., Park, M., Sim, J., Lee, J., and Lee, K. Improving audio-language learning with mixgen and multi-level test-time augmentation. CoRR, abs/2210.17143, 2022. — [arXiv:2210.17143](https://arxiv.org/abs/2210.17143) find
  44. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), Proceedings of the ICLR, 2015.
  45. Koh, J. Y., Fried, D., and Salakhutdinov, R. Generating images with multimodal language models. CoRR, abs/2305.17216, 2023. — [arXiv:2305.17216](https://arxiv.org/abs/2305.17216) find
  46. Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. CoRR, abs/2307.16125, 2023a.
  47. Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., and Liu, Z. MIMIC-IT: multi-modal in-context instruction tuning. CoRR, abs/2306.05425, 2023b. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) find
  48. Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the ICML, pp. 19730–19742, 2023c.
  49. Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. CoRR, abs/2305.06355, 2023d.
  50. Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., Kong, L., and Liu, Q. M3 it: A large-scale dataset towards multi-modal multilingual instruction tuning. CoRR, abs/2306.04387, 2023e. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) find
  51. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proceedings of the ECCV, pp. 121–137, 2020.
  52. Li, Y., Wang, X., Xiao, J., Ji, W., and Chua, T. Invariant grounding for video question answering. In Proceedings of the CVPR, pp. 2918–2927, 2022. find
  53. Li, Y., Zhang, C., Yu, G., Wang, Z., Fu, B., Lin, G., Shen, C., Chen, L., and Wei, Y. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. CoRR, abs/2308.10253, 2023f. find
  54. Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representation by alignment before projection. CoRR, abs/2311.10122, 2023.
  55. Lin, K., Li, L., Lin, C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., and Wang, L. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the CVPR, pp. 17928–17937, 2022. find
  56. Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: common objects in context. In Fleet, D. J., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Proceedings of the ECCV, pp. 740–755, 2014.
  57. Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D. P., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. In Proceedings of the ICML, pp. 21450–21474, 2023a.
  58. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. CoRR, abs/2304.08485, 2023b.
  59. Liu, S., Wang, T., Bau, D., Zhu, J., and Torralba, A. Diverse image generation via self-conditioned gans. In Proceedings of the CVPR, pp. 14274–14283, 2020. find
  60. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., and Lin, D. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023c.
  61. Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. CoRR, abs/2312.17172, 2023. — [arXiv:2312.17172](https://arxiv.org/abs/2312.17172) find
  62. Maaz, M., Rasheed, H. A., Khan, S. H., and Khan, F. S. Video-chatgpt: Towards detailed video understanding via large vision and language models. CoRR, abs/2306.05424, 2023.
  63. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the CVPR, pp. 3195–3204, 2019. find
  64. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. In Proceedings of the ICLR, 2022. find
  65. Milewski, V. S. J., Moens, M., and Calixto, I. Are scene graphs good enough to improve image captioning? In Proceedings of the AACL, pp. 504–515, 2020. find
  66. Mou, C., Wang, X., Xie, L., Zhang, J., Qi, Z., Shan, Y., and Qie, X. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. find
  67. Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the ICML, pp. 16784–16804, 2022. find
  68. OpenAI. Introducing chatgpt. 2022a. find
  69. OpenAI. Gpt-4 technical report. 2022b.
  70. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions w… — Training language models to follow instructions with human feedback
  71. Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L. V., Gross, M. H., and Sorkine-Hornung, A. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the CVPR, pp. 724–732, 2016. find
  72. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952, 2023.
  73. Qu, L., Wu, S., Fei, H., Nie, L., and Chua, T. Layoutllm-t2i: Eliciting layout guidance from LLM for text-to-image generation. In Proceedings of the ACM MM, pp. 643–654, 2023a. find
  74. Qu, L., Wu, S., Fei, H., Nie, L., and Chua, T. Layoutllm-t2i: Eliciting layout guidance from LLM for text-to-image generation. CoRR, abs/2308.05095, 2023b. — [arXiv:2308.05095](https://arxiv.org/abs/2308.05095) find
  75. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the ICML, pp. 8748–8763, 2021.
  76. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In Proceedings of the ICML, pp. 8821–8831, 2021.
  77. Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with VQ-VAE-2. In Proceedings of the NeurIPS, pp. 14837–14847, 2019. find
  78. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the CVPR, pp. 10674– 10685, 2022.
  79. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CoRR, abs/2208.12242, 2022. find
  80. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the ACL, pp. 2556–2565, 2018.
  81. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR, abs/2303.17580, 2023.
  82. Shibata, H., Hanaoka, S., Cao, Y., Yoshikawa, M., Take-naga, T., Nomura, Y., Hayashi, N., and Abe, O. Local differential privacy image generation using flow-based deep generative models. CoRR, abs/2212.10688, 2022. — [arXiv:2212.10688](https://arxiv.org/abs/2212.10688) find
  83. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-a-video: Text-to-video generation without text-video data. CoRR, abs/2209.14792, 2022. find
  84. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Proceedings of the NeurIPS, 2020. find
  85. Su, Y., Lan, T., Liu, Y., Liu, F., Yogatama, D., Wang, Y., Kong, L., and Collier, N. Language models can see: Plugging visual controls in text generation. CoRR, abs/2205.02655, 2022. — [arXiv:2205.02655](https://arxiv.org/abs/2205.02655) find
  86. Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. CoRR, abs/2305.16355, 2023.
  87. Sun, Q., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., and Wang, X. Generative pretraining in multimodality. CoRR, abs/2307.05222, 2023. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) find
  88. Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. Any-to-any generation via composable diffusion. CoRR, abs/2305.11846, 2023. find
  89. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. 2023. URL https://github.com/tatsu-lab/stanford_alpaca. ↗
  90. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
  91. Vahdat, A. and Kautz, J. NVAE: A deep hierarchical variational autoencoder. In Proceedings of the NeurIPS, 2020. find
  92. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the NeurIPS, pp. 5998– 6008, 2017.
  93. Veaux, C., Yamagishi, J., MacDonald, K., et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. CSTR, 6:15, 2017. find
  94. Voynov, A., Chu, Q., Cohen-Or, D., and Aberman, K. P+: extended textual conditioning in text-to-image generation. CoRR, abs/2303.09522, 2023. — [arXiv:2303.09522](https://arxiv.org/abs/2303.09522) find
  95. Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. GIT: A generative image-to-text transformer for vision and language. Trans. Mach. Learn. Res., 2022, 2022a.
  96. Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Proceedings of the ICML, volume 162, 2022b.
  97. Wang, T., Yi, J., Fu, R., Tao, J., and Wen, Z. Campnet: Context-aware mask prediction for end-to-end text-based speech editing. IEEE ACM Trans. Audio Speech Lang. Process., 30:2241–2254, 2022c. find
  98. Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023.
  99. Wu, J. Z., Ge, Y., Wang, X., Lei, W., Gu, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. CoRR, abs/2212.11565, 2022. find
  100. Xiao, J., Shang, X., Yao, A., and Chua, T. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the CVPR, pp. 9777–9786, 2021. find
  101. Xiao, J., Yao, A., Liu, Z., Li, Y., Ji, W., and Chua, T. Video as conditional graph hierarchy for multi-granular question answering. In Proceedings of the AAAI, pp. 2804–2812, 2022. find
  102. Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., and Zhuang, Y. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the ACM MM, pp. 1645–1653, 2017. find
  103. Xu, H., Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., Wang, W., Xu, G., Zhang, J., Huang, S., Huang, F., and Zhou, J. mplug-2: A modularized multimodal foundation model across text, image and video. In Proceedings of the ICML, pp. 38728–38748, 2023. find
  104. Xu, J., Mei, T., Yao, T., and Rui, Y. MSR-VTT: A large video description dataset for bridging video and language. In Proceedings of the CVPR, pp. 5288–5296, 2016. find
  105. Xu, J., Mello, S. D., Liu, S., Byeon, W., Breuel, T. M., Kautz, J., and Wang, X. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the CVPR, pp. 18113–18123, 2022. find
  106. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the CVPR, pp. 1316–1324, 2018. find
  107. Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the ICCV, pp. 1666– 1677, 2021. find
  108. Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:1720–1733, 2023. find
  109. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., Qi, Q., Zhang, J., and Huang, F. mplug-docowl: Modularized multimodal large language model for document understanding. CoRR, abs/2307.02499, 2023a. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) find
  110. Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., Li, C., Xu, Y., Chen, H., Tian, J., Qi, Q., Zhang, J., and Huang, F. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/2304.14178, 2023b.
  111. Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Sheng, L., Bai, L., Huang, X., Wang, Z., Shao, J., and Ouyang, W. LAMM: language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. CoRR, abs/2306.06687, 2023. find
  112. Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67–78, 2014. find
  113. Yu, B., Fu, C., Yu, H., Huang, F., and Li, Y. Unified language representation for question answering over text, tables, and images. CoRR, abs/2306.16762, 2023. find
  114. Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., and Wang, Z. Conzic: Controllable zero-shot image captioning by sampling-based polishing. In Proceedings of the CVPR, pp. 23465–23476, 2023. find
  115. Zhang, A., Fei, H., Yao, Y., Ji, W., Li, L., Liu, Z., and Chua, T. Transfer visual prompt generator across llms. CoRR, abs/2305.01278, 2023a. find
  116. Zhang, B., Gu, S., Zhang, B., Bao, J., Chen, D., Wen, F., Wang, Y., and Guo, B. Styleswin: Transformer-based GAN for high-resolution image generation. In Proceedings of the CVPR, pp. 11294–11304, 2022. find
  117. Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. CoRR, abs/2305.11000, 2023b.
  118. Zhang, H., Li, X., and Bing, L. Video-llama: An instruction-tuned audio-visual language model for video understanding. CoRR, abs/2306.02858, 2023c.
  119. Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding. CoRR, abs/2306.17107, 2023d.
  120. Zhang, Z., Shi, Y., Yuan, C., Li, B., Wang, P., Hu, W., and Zha, Z. Object relational graph with teacher-recommended learning for video captioning. In Proceedings of the CVPR, pp. 13275–13285, 2020. find
  121. Zhao, B., Wu, B., and Huang, T. SVIT: scaling up visual instruction tuning. CoRR, abs/2307.04087, 2023a. — [arXiv:2307.04087](https://arxiv.org/abs/2307.04087) find
  122. Zhao, L., Yu, E., Ge, Z., Yang, J., Wei, H., Zhou, H., Sun, J., Peng, Y., Dong, R., Han, C., and Zhang, X. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. CoRR, abs/2307.09474, 2023b. — [arXiv:2307.09474](https://arxiv.org/abs/2307.09474) find
  123. Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal llms. CoRR, abs/2307.08581, 2023c. — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) find
  124. Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., and Gao, J. Regionclip: Region-based language-image pretraining. In Proceedings of the CVPR, pp. 16772–16782, 2022. find
  125. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592, 2023.
  126. Zhu, M., Pan, P., Chen, W., and Yang, Y. DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the CVPR, pp. 5802–5810, 2019. find
  127. A. Potential Limitation and Future work As future work, there are at least following four avenues to explore. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-next-gpt-any-any-multimodal/