Elamur Research⌕
☰
home / papers / 2023 / V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

Paper · 2023 · Penghao Wu and Saining Xie

Summary

Multimodal LLMs usually view images through frozen vision encoders such as CLIP, which work at low resolution (e.g., 224×224 or 336×336). As a result, small details in large or cluttered pictures are lost. The models also cannot notice what is missing or go looking for it. Drawing on how people search scenes, the authors propose SEAL (Show, sEArch, and TelL), a meta-architecture that pairs a VQA LLM with a visual search model.

When the VQA LLM finds the visible evidence insufficient, it names the objects it needs. V*, a search procedure driven by a second MLLM, then locates them by zooming into the image hierarchically. It is guided by common-sense contextual cues (where such an object tends to be) and target-specific cues. Located regions go into a Visual Working Memory, which is fed back to the VQA LLM for answering. Both parts build on LLaVA with Vicuna-7B.

The authors also introduce V* Bench, a VQA benchmark for fine-grained grounding in high-resolution images. Most tested MLLMs score near chance on it, while SEAL improves strongly over its baseline. This suggests active visual search should be a core MLLM capability.

Key points

  • On V* Bench, SEAL (7B) reaches 75.3 versus 45.0 for a comparable LLaVA (7B) trained on the same VQA data. Several MLLMs, among them GPT-4V, Bard and Gemini Pro, were evaluated, and most scored roughly at random-guess level.
  • On general benchmarks, SEAL improves MME (1051.2 → 1128.9) and the hallucination benchmark POPE (76.5 → 82.4). It stays about level on MMBench and SEED-Bench and drops slightly on MM-Vet and LLaVA-Bench W.
  • Search efficiency was tested on the 245 annotated target locations in V* Bench. V* needs far fewer search steps than Random and Sequential baselines (each run in BFS and DFS modes), and both contextual and target cues contribute.
  • On COCO-Search18, the authors guided the search with heatmaps built from human eye fixations. V* proved roughly as efficient as this human-fixation-guided search.
  • Stated limitations: the search model is tuned for natural images and common objects. Documents, diagrams, long videos and open-world settings would require further training and new algorithm design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States219 works · 103,487 citations🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Penghao Wu and Saining Xie. 2023. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. arXiv:2312.14135.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.14135 · arXiv 2312.14135

@misc{wu2023v,
  title = {{V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs}},
  author = {Penghao Wu and Saining Xie},
  year = {2023},
  doi = {10.48550/arxiv.2312.14135},
  eprint = {2312.14135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.14135}
}

References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew B… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020. 7
  3. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 13 find
  4. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 13
  5. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 4
  6. Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In CVPR, 2022. 6 find
  7. Yupei Chen, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco-search18 fixation dataset for predicting goal-directed attention control. Scientific reports, 2021. 8, 9 find
  8. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023. 2, 3, 4, 7
  9. Google DeepMind. Gemini, 2023. 7 find
  10. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2, 9
  11. Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023. 7
  12. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 14
  13. Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language models. In NeurIPS, 2023. 4 [46] Antonio Torralba, Aude Oliva, Monica S Castelhano, and John M Henderson. Contextual guidance…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/