Elamur Research⌕
☰
home / papers / 2023 / PandaGPT: One Model To Instruction-Follow Them All

PandaGPT: One Model To Instruction-Follow Them All

Paper · 2023 · Yixuan Su, Tian Lan, Huayang Li et al.

Summary

Most multimodal instruction-following systems pair text with only one other input type, such as images or speech, so they cannot reason jointly over several kinds of signals. PandaGPT addresses this by attaching the frozen multimodal encoders of ImageBind to the frozen Vicuna-13B language model. Only two small components are learned: a linear layer that maps ImageBind embeddings into Vicuna's input space, and LoRA adapters inserted into Vicuna's attention layers. Training uses roughly 160k image-grounded multi-turn conversations from the LLaVA and MiniGPT-4 releases, with the loss applied only to model responses. ImageBind places six modalities in a common embedding space. As a result, the model is tuned on image-text data alone yet responds zero-shot to video, audio, depth, thermal and IMU inputs, and it can combine them by adding their embeddings. The evidence is qualitative: the authors show visual question answering, story writing from videos, sound-based reasoning and cross-modal "arithmetic," but report no benchmark scores. The work matters as a cheap, reusable recipe showing how a shared embedding space can extend an LLM to many senses at once.

Key points

  • Architecture: ImageBind encoders (frozen), a linear projection, and Vicuna-13B (version 0, frozen) with LoRA weights. The trainable parameters are only about 0.4% of Vicuna's size.
  • Training setup: 2 epochs, learning rate 5e-4 with linear decay, maximum sequence length 400, on 8×A100 40G GPUs, taking about 7 hours.
  • The model is claimed to handle six input types in total: text, image/video, audio, depth, thermal readings and IMU signals. Everything beyond image-text is emergent, with no paired training data for those modalities.
  • Each non-text input is compressed into a single embedding vector. The authors list finer-grained fusion, such as cross-modal attention, as future work.
  • Stated limitations:
    • The model accepts multimodal input but produces text only.
    • New benchmarks are needed to test how well it composes multiple modalities.
    • It inherits LLM problems such as hallucination, toxicity and stereotypes.
    • The authors present it as a research prototype, not a production-ready tool.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🎓Nara Institute of Science and TechnologyIkoma · Japan34 works · 4,419 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 269 works

Cite

Cite as Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang and Deng Cai. 2023. PandaGPT: One Model To Instruction-Follow Them All. arXiv:2305.16355.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.16355 · arXiv 2305.16355

@misc{su2023pandagpt,
  title = {{PandaGPT: One Model To Instruction-Follow Them All}},
  author = {Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang and Deng Cai},
  year = {2023},
  doi = {10.48550/arxiv.2305.16355},
  eprint = {2305.16355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.16355}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736.
  2. Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. 2020. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25–37. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877– 1901.
  4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612. find
  7. Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26. find
  8. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305.05665.
  9. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976–980. IEEE. find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  11. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  12. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.
  13. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. arXiv preprint arXiv:2104.08773, pages 839–849. find
  14. OpenAI. 2022. Gpt-4 technical report.
  15. OpenAI. 2022. Introducing chatgpt. find
  16. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  17. Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Lingpeng Kong, and Tong Zhang. 2023. Detgpt: Detect what you need via reasoning. find
  18. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM…
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. find
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  21. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
  22. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  23. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021. find
  24. Yixuan Su, Tian Lan, and Deng Cai. 2023. Openalpaca: A fully open-source instruction-following model based on openllama. https://github.com/yxuansu/OpenAlpaca. ↗
  25. Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. 2022. Language models can see: plugging visual controls in text generation. arXiv preprint arXiv:2205.02655. — [arXiv:2205.02655](https://arxiv.org/abs/2205.02655) find
  26. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  28. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. find
  29. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.
  30. Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction-finetuned visual language model for video understanding. find
  31. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/