Elamur Research⌕
☰
home / papers / 2023 / Vision-Language Models for Vision Tasks: A Survey

Vision-Language Models for Vision Tasks: A Survey

Paper · 2023 · Jingyi Zhang, Jiaxing Huang, Sheng Jin et al.

Summary

This survey, by researchers at Nanyang Technological University, addresses two drawbacks of mainstream visual recognition: dependence on costly crowd-labelled data and the habit of training a separate network for every task. It follows the shift toward vision-language models (VLMs), which learn links between images and text from huge web-crawled image-text pairs. A single such model can then handle many recognition tasks zero-shot, without any fine-tuning. The authors trace five stages of training paradigms, ending with VLM pre-training plus zero-shot prediction, pioneered by CLIP in 2021. They then cover common architectures, pre-training objectives and frameworks, and the datasets used for pre-training and evaluation. Existing work is sorted into three families: pre-training methods, transfer learning, and knowledge distillation into detection and segmentation models. The survey benchmarks the reviewed methods on public datasets and closes with open problems. It is a structured entry point to a field whose publication count has grown exponentially since CLIP.

Key points

  • CLIP uses an image-text contrastive objective, which brings matching image and caption embeddings together and separates mismatched ones. It achieved strong zero-shot results across 36 visual recognition tasks.
  • The paper describes transfer approaches, including prompt tuning and visual adaptation, and distillation approaches for object detection and semantic segmentation.
  • The evaluation datasets span several task types:
    • Classification: ImageNet-1k, SVHN, IIIT5k, and Rendered SST2 (built from Stanford Sentiment Treebank sentences).
    • Action recognition: UCF101, Kinetics700, RareAct.
    • Segmentation: ADE20k (150 classes), PASCAL VOC 2012, Cityscapes.
    • Detection: MS COCO, LVIS (1203 categories), ODinW.
    • Retrieval: Flickr30k, COCO Caption.
  • ODinW combines 35 public detection datasets covering 314 concepts, with about 132k training and 20k test images.
  • A companion repository is maintained at github.com/jingyi0000/VLM survey.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Jingyi Zhang, Jiaxing Huang, Sheng Jin and Shijian Lu. 2023. Vision-Language Models for Vision Tasks: A Survey. arXiv:2304.00685.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.00685 · arXiv 2304.00685

@misc{zhang2023vision,
  title = {{Vision-Language Models for Vision Tasks: A Survey}},
  author = {Jingyi Zhang, Jiaxing Huang, Sheng Jin and Shijian Lu},
  year = {2023},
  doi = {10.48550/arxiv.2304.00685},
  eprint = {2304.00685},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.00685}
}

References

205 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Geiger et al. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pp. 3354–3361. IEEE, 2012. find
  2. G. Cheng et al. Remote sensing image scene classification meets deep learning: Challenges, methods, benchmarks, and opportunities. JSTARS, 13:3735–3756, 2020. find
  3. H. A. Pierson and M. S. Gashler. Deep learning in robotics: a review of recent research. Advanced Robotics, 31(16):821–835, 2017. find
  4. A. Krizhevsky et al. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017. find
  5. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  6. K. He et al. Deep residual learning for image recognition. In CVPR, pp. 770–778, 2016.
  7. L. E. Peterson. K-nearest neighbor. Scholarpedia, 4(2):1883, 2009. find
  8. C. Cortes and V. Vapnik. Support-vector networks. Machine learning, 20:273–297, 1995.
  9. A. Mathur and G. M. Foody. Multiclass and binary svm classification: Implications for training and classification users. IEEE Geoscience and remote sensing letters, 5(2):241–245, 2008. find
  10. A. Radford et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748–8763. PMLR, 2021.
  11. R. Girshick. Fast r-cnn. In ICCV, pp. 1440–1448, 2015. find
  12. K. He et al. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729–9738, 2020.
  13. T. Chen et al. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597–1607. PMLR, 2020.
  14. J. Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  15. A. Radford et al. Improving language understanding by generative pre-training. 2018. find
  16. A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  17. C. Jia et al. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp. 4904–4916. PMLR, 2021.
  18. L. Yao et al. Filip: Fine-grained interactive language-image pre-training. In ICLR, 2021. find
  19. J. Yu et al. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  20. C. Schuhmann et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
  21. C. Schuhmann et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  22. L. Bossard et al. Food-101–mining discriminative components with random forests. In ECCV, pp. 446–461. Springer, 2014. find
  23. A. Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. find
  24. J. Xiao et al. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010. find
  25. J. Krause et al. Collecting a large-scale dataset of fine-grained cars. 2013. find
  26. O. M. Parkhi et al. Cats and dogs. In CVPR, pp. 3498–3505. IEEE, 2012. find
  27. D. Kiela et al. The hateful memes challenge: Detecting hate speech in multimodal memes. NeurIPS, 33:2611–2624, 2020.
  28. A. Miech et al. Rareact: A video dataset of unusual interactions. arXiv preprint arXiv:2008.01018, 2020. find
  29. K. Soomro et al. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  30. J. Carreira et al. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. find
  31. K. Zhou et al. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022. find
  32. K. Zhou et al. Conditional prompt learning for vision-language models. In CVPR, pp. 16816–16825, 2022. find
  33. P. Gao et al. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021. find
  34. R. Zhang et al. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021. find
  35. J. Ding et al. Decoupling zero-shot semantic segmentation. In CVPR, pp. 11583–11592, 2022. find
  36. X. Gu et al. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2021. find
  37. Y. Du et al. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, pp. 14084– 14093, 2022. find
  38. D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60:91–110, 2004. find
  39. L. Breiman. Random forests. Machine learning, 45:5–32, 2001. find
  40. J. Deng et al. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248–255. Ieee, 2009. find
  41. K. He et al. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000–16009, 2022.
  42. A. Singh et al. Flava: A foundational language and vision alignment model. In CVPR, pp. 15638–15650, 2022. find
  43. M. Tschannen et al. Image-and-language understanding from pixels only. arXiv preprint arXiv:2212.08045, 2022. — [arXiv:2212.08045](https://arxiv.org/abs/2212.08045) find
  44. J. Jang et al. Unifying vision-language representation space with single-tower transformer. In AAAI, volume 37, pp. 980–988, 2023. find
  45. L. Yao et al. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. In NeurIPS. find
  46. H. Luo et al. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. arXiv preprint arXiv:2211.14813, 2022. — [arXiv:2211.14813](https://arxiv.org/abs/2211.14813) find
  47. S. Antol et al. Vqa: Visual question answering. In ICCV, pp. 2425–2433, 2015. find
  48. A. Suhr et al. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018.
  49. A. Karpathy et al. Deep fragment embeddings for bidirectional image sentence mapping. NeurIPS, 27, 2014. find
  50. F. Li et al. Vision-language intelligence: Tasks, representation learning, and large models. arXiv preprint arXiv:2203.01922, 2022. — [arXiv:2203.01922](https://arxiv.org/abs/2203.01922) find
  51. Y. Du et al. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936, 2022. — [arXiv:2202.10936](https://arxiv.org/abs/2202.10936) find
  52. F.-L. Chen et al. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38–56, 2023. find
  53. P. Xu et al. Multimodal learning with transformers: A survey. arXiv preprint arXiv:2206.06488, 2022. — [arXiv:2206.06488](https://arxiv.org/abs/2206.06488) find
  54. X. Wang et al. Large-scale multi-modal pre-trained models: A comprehensive survey. arXiv preprint arXiv:2302.10035, 2023. — [arXiv:2302.10035](https://arxiv.org/abs/2302.10035) find
  55. S. Ren et al. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 28, 2015.
  56. L.-C. Chen et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834–848, 2017. find
  57. A. Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  58. A. Vaswani et al. Attention is all you need. NeurIPS, 30, 2017.
  59. M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pp. 6105–6114. PMLR, 2019.
  60. T. He et al. Bag of tricks for image classification with convolutional neural networks. In CVPR, pp. 558–567, 2019. find
  61. R. Zhang. Making convolutional networks shift-invariant again. In ICML, pp. 7324–7334. PMLR, 2019. find
  62. N. Carion et al. End-to-end object detection with transformers. In ECCV, pp. 213–229. Springer, 2020.
  63. E. Xie et al. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34:12077–12090, 2021. find
  64. N. Mu et al. Slip: Self-supervision meets language-image pre-training. In ECCV, pp. 529–544. Springer, 2022. find
  65. J. Yang et al. Unified contrastive learning in image-text-label space. In CVPR, pp. 19163–19173, 2022. find
  66. K. He et al. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
  67. L. H. Li et al. Grounded language-image pre-training. In CVPR, pp. 10965–10975, 2022. find
  68. A. v. d. Oord et al. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  69. P. Khosla et al. Supervised contrastive learning. NeurIPS, 33:18661–18673, 2020. find
  70. H. Bao et al. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
  71. Z.-Y. Dou et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. In NeurIPS. find
  72. H. Bao et al. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358, 2021. find
  73. V. Ordonez et al. Im2text: Describing images using 1 million captioned photographs. NeurIPS, 24, 2011. find
  74. X. Chen et al. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  75. B. Thomee et al. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
  76. R. Krishna et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32– 73, 2017.
  77. P. Sharma et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556–2565, 2018.
  78. J. Pont-Tuset et al. Connecting vision and language with localized narratives. In ECCV, pp. 647–664. Springer, 2020. find
  79. S. Changpinyo et al. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558–3568, 2021. find
  80. K. Srinivasan et al. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In ACM SIGIR, pp. 2443–2449, 2021.
  81. K. Desai et al. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431, 2021. — [arXiv:2111.11431](https://arxiv.org/abs/2111.11431) find
  82. J. Gu et al. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework. arXiv preprint arXiv:2202.06767, 2022. — [arXiv:2202.06767](https://arxiv.org/abs/2202.06767) find
  83. X. Chen et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
  84. J. Lee et al. Uniclip: Unified framework for contrastive language-image pre-training. In NeurIPS. find
  85. S. Shao et al. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, pp. 8430–8439, 2019. find
  86. A. Kamath et al. Mdetr-modulated detection for end-to-end multi-modal understanding. In ICCV, pp. 1780–1790, 2021. find
  87. M. Cao et al. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.14713, 2022. — [arXiv:2203.14713](https://arxiv.org/abs/2203.14713) find
  88. Y. LeCun et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  89. L. Fei-Fei et al. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshop, pp. 178–178. IEEE, 2004. find
  90. M. Everingham et al. The pascal visual object classes (voc) challenge. IJCV, 88(2):303–338, 2010. find
  91. M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 722–729. IEEE, 2008. find
  92. Y. Netzer et al. Reading digits in natural images with unsupervised feature learning. 2011. find
  93. A. Coates et al. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, pp. 215–223. JMLR Workshop and Conference Proceedings, 2011. find
  94. J. Stallkamp et al. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, pp. 1453–1460. IEEE, 2011. find
  95. A. Mishra et al. Scene text recognition using higher order language priors. In BMVC-British machine vision conference. BMVA, 2012. find
  96. S. Maji et al. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. — [arXiv:1306.5151](https://arxiv.org/abs/1306.5151) find
  97. I. J. Goodfellow et al. Challenges in representation learning: A report on three machine learning contests. In ICONIP, pp. 117– 124. Springer, 2013. find
  98. R. Socher et al. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631–1642, 2013.
  99. M. Cimpoi et al. Describing textures in the wild. In CVPR, pp. 3606–3613, 2014. find
  100. T. Berg et al. Birdsnap: Large-scale fine-grained visual categorization of birds. In CVPR, pp. 2011–2018, 2014. find
  101. G. Cheng et al. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865– 1883, 2017. find
  102. J. Johnson et al. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 2901– 2910, 2017.
  103. B. S. Veeling et al. Rotation equivariant cnns for digital pathology. In International Conference on Medical image computing and computer-assisted intervention, pp. 210–218. Springer, 2018. find
  104. P. Helber et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JSTARS, 12(7):2217–2226, 2019. find
  105. P. Young et al. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67–78, 2014. find
  106. T.-Y. Lin et al. Microsoft coco: Common objects in context. In ECCV, pp. 740–755. Springer, 2014.
  107. A. Gupta et al. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, pp. 5356–5364, 2019. find
  108. C. Li et al. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. arXiv preprint arXiv:2204.08790, 2022. — [arXiv:2204.08790](https://arxiv.org/abs/2204.08790) find
  109. R. Mottaghi et al. The role of context for object detection and semantic segmentation in the wild. In CVPR, pp. 891–898, 2014. find
  110. M. Cordts et al. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213–3223, 2016. find
  111. B. Zhou et al. Scene parsing through ade20k dataset. In CVPR, pp. 633–641, 2017. find
  112. B. Wu et al. Data efficient language-supervised zero-shot recognition with optimal transport distillation. In ICLR, 2021. find
  113. Y. Li et al. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2021. find
  114. Q. Cui et al. Contrastive vision-language pre-training with limited resources. In ECCV, pp. 236–253. Springer, 2022. find
  115. L. Yuan et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. — [arXiv:2111.11432](https://arxiv.org/abs/2111.11432) find
  116. Y. Gao et al. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining. arXiv preprint arXiv:2204.14095, 2022. — [arXiv:2204.14095](https://arxiv.org/abs/2204.14095) find
  117. A. Yang et al. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335, 2022. — [arXiv:2211.01335](https://arxiv.org/abs/2211.01335) find
  118. X. Zhai et al. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pp. 18123–18133, 2022. find
  119. Z. Chen et al. Altclip: Altering the language encoder in clip for extended language capabilities. arXiv preprint arXiv:2211.06679, 2022. find
  120. B. Ko and G. Gu. Large-scale bilingual language-image contrastive learning. arXiv preprint arXiv:2203.14463, 2022. — [arXiv:2203.14463](https://arxiv.org/abs/2203.14463) find
  121. J. Zhou et al. Non-contrastive learning meets language-image pre-training. arXiv preprint arXiv:2210.09304, 2022. — [arXiv:2210.09304](https://arxiv.org/abs/2210.09304) find
  122. S. Shen et al. K-lite: Learning transferable visual models with external knowledge. arXiv preprint arXiv:2204.09222, 2022. find
  123. R. Huang et al. Nlip: Noise-robust language-image pre-training. arXiv preprint arXiv:2212.07086, 2022. — [arXiv:2212.07086](https://arxiv.org/abs/2212.07086) find
  124. S. Geng et al. Hiclip: Contrastive language-image pretraining with hierarchy-aware attention. arXiv preprint arXiv:2303.02995, 2023. — [arXiv:2303.02995](https://arxiv.org/abs/2303.02995) find
  125. C.-W. Xie et al. Ra-clip: Retrieval augmented contrastive language-image pre-training. In CVPR, pp. 19265–19274, 2023. find
  126. L. Fan et al. Improving clip training with language rewrites. arXiv preprint arXiv:2305.20088, 2023. — [arXiv:2305.20088](https://arxiv.org/abs/2305.20088) find
  127. K. Yang et al. Alip: Adaptive language-image pre-training with synthetic caption. In ICCV, pp. 2922–2931, 2023. find
  128. X. Deng et al. Growclip: Data-aware automatic model growing for large-scale contrastive language-image pre-training. In ICCV, pp. 22178–22189, 2023. find
  129. J. Xu et al. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pp. 18134–18144, 2022. find
  130. K. Ranasinghe et al. Perceptual grouping in vision-language models. arXiv preprint arXiv:2210.09996, 2022. — [arXiv:2210.09996](https://arxiv.org/abs/2210.09996) find
  131. Y. Zhong et al. Regionclip: Region-based language-image pre-training. In CVPR, pp. 16793–16803, 2022. find
  132. C. Ma et al. Understanding and mitigating overfitting in prompt tuning for vision-language models. arXiv preprint arXiv:2211.02219, 2022. — [arXiv:2211.02219](https://arxiv.org/abs/2211.02219) find
  133. A. Bulat and G. Tzimiropoulos. Language-aware soft prompting for vision & language foundation models. arXiv preprint arXiv:2210.01115, 2022. — [arXiv:2210.01115](https://arxiv.org/abs/2210.01115) find
  134. Y. Lu et al. Prompt distribution learning. In CVPR, pp. 5206–5215, 2022. find
  135. M. M. Derakhshani et al. Variational prompt tuning improves generalization of vision-language models. arXiv preprint arXiv:2210.02390, 2022. — [arXiv:2210.02390](https://arxiv.org/abs/2210.02390) find
  136. B. Zhu et al. Prompt-aligned gradient for prompt tuning. arXiv preprint arXiv:2205.14865, 2022. find
  137. X. He et al. Cpl: Counterfactual prompt learning for vision and language models. arXiv preprint arXiv:2210.10362, 2022. find
  138. G. Chen et al. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022. — [arXiv:2210.01253](https://arxiv.org/abs/2210.01253) find
  139. X. Sun et al. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurIPS. find
  140. Z. Guo et al. Texts as images in prompt tuning for multi-label image recognition. arXiv preprint arXiv:2211.12739, 2022. — [arXiv:2211.12739](https://arxiv.org/abs/2211.12739) find
  141. K. Ding et al. Prompt tuning with soft context sharing for vision-language models. arXiv preprint arXiv:2208.13474, 2022. — [arXiv:2208.13474](https://arxiv.org/abs/2208.13474) find
  142. Y. Rao et al. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, pp. 18082–18091, 2022. find
  143. T. Huang et al. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. — [arXiv:2204.03649](https://arxiv.org/abs/2204.03649) find
  144. M. Shu et al. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS. find
  145. H. Yao et al. Visual-language prompt tuning with knowledge-guided context optimization. In CVPR, pp. 6757–6767, 2023. find
  146. T.-Y. Wu et al. Protect: Prompt tuning for hierarchical consistency. arXiv preprint arXiv:2306.02240, 2023. — [arXiv:2306.02240](https://arxiv.org/abs/2306.02240) find
  147. H. Bahng et al. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 1(3):4, 2022. — [arXiv:2203.17274](https://arxiv.org/abs/2203.17274) find
  148. J. Rong et al. Retrieval-enhanced visual prompt learning for few-shot classification. arXiv preprint arXiv:2306.02243, 2023. — [arXiv:2306.02243](https://arxiv.org/abs/2306.02243) find
  149. Y. Zang et al. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022. — [arXiv:2210.07225](https://arxiv.org/abs/2210.07225) find
  150. S. Shen et al. Multitask vision-language prompt tuning. arXiv preprint arXiv:2211.11720, 2022. find
  151. M. U. Khattak et al. Maple: Multi-modal prompt learning. arXiv preprint arXiv:2210.03117, 2022. find
  152. Y. Xing et al. Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:2208.08340, 2022. — [arXiv:2208.08340](https://arxiv.org/abs/2208.08340) find
  153. O. Pantazis et al. Svl-adapter: Self-supervised adapter for vision-language pretrained models. arXiv preprint arXiv:2210.03794, 2022. — [arXiv:2210.03794](https://arxiv.org/abs/2210.03794) find
  154. V. Udandarao et al. Sus-x: Training-free name-only transfer of vision-language models. arXiv preprint arXiv:2211.16198, 2022. — [arXiv:2211.16198](https://arxiv.org/abs/2211.16198) find
  155. J. Kahana et al. Improving zero-shot models with label distribution priors. arXiv preprint arXiv:2212.00784, 2022. — [arXiv:2212.00784](https://arxiv.org/abs/2212.00784) find
  156. F. Peng et al. Sgva-clip: Semantic-guided visual adapting of vision-language models for few-shot image classification. arXiv preprint arXiv:2211.16191, 2022. — [arXiv:2211.16191](https://arxiv.org/abs/2211.16191) find
  157. R. Zhang et al. Vt-clip: Enhancing vision-language models with visual-guided texts. arXiv preprint arXiv:2112.02399, 2021. — [arXiv:2112.02399](https://arxiv.org/abs/2112.02399) find
  158. Z. Guo et al. Calip: Zero-shot enhancement of clip with parameter-free attention. arXiv preprint arXiv:2209.14169, 2022. — [arXiv:2209.14169](https://arxiv.org/abs/2209.14169) find
  159. T. Yu et al. Task residual for tuning vision-language models. In CVPR, pp. 10899–10909, 2023. find
  160. S. Pratt et al. What does a platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.03320, 2022. find
  161. S. Menon and C. Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022. — [arXiv:2210.07183](https://arxiv.org/abs/2210.07183) find
  162. M. Wortsman et al. Robust fine-tuning of zero-shot models. In CVPR, pp. 7959–7971, 2022. find
  163. C. Zhou et al. Extract free dense labels from clip. In ECCV, pp. 696–712. Springer, 2022. find
  164. J. Li et al. Masked unsupervised self-training for zero-shot image classification. arXiv preprint arXiv:2206.02967, 2022. — [arXiv:2206.02967](https://arxiv.org/abs/2206.02967) find
  165. P. Liu et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  166. M. Jia et al. Visual prompt tuning. arXiv preprint arXiv:2203.12119, 2022. — [arXiv:2203.12119](https://arxiv.org/abs/2203.12119) find
  167. N. Houlsby et al. Parameter-efficient transfer learning for nlp. In ICML, pp. 2790–2799. PMLR, 2019. find
  168. R. Zhang et al. Pointclip: Point cloud understanding by clip. In CVPR, pp. 8552–8562, 2022. find
  169. T. Huang et al. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, pp. 22157–22167, 2023. find
  170. M. Xu et al. Side adapter network for open-vocabulary semantic segmentation. In CVPR, pp. 2945–2954, 2023. find
  171. G. Chen et al. Tem-adapter: Adapting image-text pretraining for video question answer. In ICCV, pp. 13945–13955, 2023. find
  172. T. Brown et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020.
  173. Y. Zang et al. Open-vocabulary detr with conditional matching. arXiv preprint arXiv:2203.11876, 2022. — [arXiv:2203.11876](https://arxiv.org/abs/2203.11876) find
  174. Z. Zhou et al. Zegclip: Towards adapting clip for zero-shot semantic segmentation. arXiv preprint arXiv:2212.03588, 2022. find
  175. T. Lüddecke and A. Ecker. Image segmentation using text and image prompts. In CVPR, pp. 7086–7096, 2022. find
  176. B. Li et al. Language-driven semantic segmentation. In ICLR, 2021. find
  177. N. Zabari and Y. Hoshen. Semantic segmentation in-the-wild without seeing any segmentation examples. arXiv preprint arXiv:2112.03185, 2021. — [arXiv:2112.03185](https://arxiv.org/abs/2112.03185) find
  178. C. Ma et al. Open-vocabulary semantic segmentation with frozen vision-language models. arXiv preprint arXiv:2210.15138, 2022. — [arXiv:2210.15138](https://arxiv.org/abs/2210.15138) find
  179. F. Liang et al. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150, 2022. find
  180. M. Xu et al. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In ECCV, pp. 736–753. Springer, 2022. find
  181. G. Ghiasi et al. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, pp. 540–557. Springer, 2022. find
  182. G. Shin et al. Reco: Retrieve and co-segment for zero-shot transfer. In NeurIPS. find
  183. J. Xie et al. Clims: Cross language image matching for weakly supervised semantic segmentation. In CVPR, pp. 4483–4492, 2022. find
  184. Y. Lin et al. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. arXiv preprint arXiv:2212.09506, 2022. find
  185. J. Qin et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In CVPR, pp. 19446–19455, 2023. find
  186. Z. Ma et al. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In CVPR, pp. 14074– 14083, 2022. find
  187. H. A. Rasheed et al. Bridging the gap between object and image-level representations for open-vocabulary detection. In NeurIPS. find
  188. C. Feng et al. Promptdet: Towards open-vocabulary detection using uncurated images. In ECCV, pp. 701–717. Springer, 2022. find
  189. M. Gao et al. Open vocabulary object detection with pseudo bounding-box labels. In ECCV, pp. 266–282. Springer, 2022. find
  190. T. Wang and N. Li. Learning to detect and segment for open vocabulary object detection. arXiv preprint arXiv:2212.12130, 2022. — [arXiv:2212.12130](https://arxiv.org/abs/2212.12130) find
  191. C. Lin et al. Learning object-language alignments for open-vocabulary object detection. arXiv preprint arXiv:2211.14843, 2022. — [arXiv:2211.14843](https://arxiv.org/abs/2211.14843) find
  192. W. Kuo et al. F-vlm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15639, 2022. — [arXiv:2209.15639](https://arxiv.org/abs/2209.15639) find
  193. X. Zhou et al. Detecting twenty-thousand classes using image-level supervision. In ECCV, pp. 350–368. Springer, 2022. find
  194. D. Huynh et al. Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling. In CVPR, pp. 7020–7031, 2022. find
  195. M. Minderer et al. Simple open-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.06230, 2022. — [arXiv:2205.06230](https://arxiv.org/abs/2205.06230) find
  196. S. Zhao et al. Exploiting unlabeled data with vision and language models for object detection. In ECCV, pp. 159–175. Springer, 2022. find
  197. Y. Long et al. P3ovd: Fine-grained visual-text prompt-driven self-training for open-vocabulary object detection. arXiv preprint arXiv:2211.00849, 2022. — [arXiv:2211.00849](https://arxiv.org/abs/2211.00849) find
  198. J. Xie and S. Zheng. Zsd-yolo: Zero-shot yolo detection using vision-language knowledgedistillation. arXiv preprint arXiv:2109.12066, 2021. — [arXiv:2109.12066](https://arxiv.org/abs/2109.12066) find
  199. D. Kim et al. Region-aware pretraining for open-vocabulary object detection with vision transformers. In CVPR, pp. 11144– 11154, 2023. find
  200. S. Wu et al. Aligning bag of regions for open-vocabulary object detection. In CVPR, pp. 15254–15264, 2023. find
  201. L. Wang et al. Object-aware distillation pyramid for open-vocabulary object detection. In CVPR, pp. 11186–11196, 2023. find
  202. M. Cherti et al. Reproducible scaling laws for contrastive language-image learning. In CVPR, pp. 2818–2829, 2023.
  203. Y. Xian et al. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, pp. 8256–8265, 2019. find
  204. X. Zhai et al. Scaling vision transformers. In CVPR, pp. 12104– 12113, 2022. find
  205. C. Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020. A PPENDIX A. S TATISTICS ON V ISUAL R ECOGNITION VLM P UBLICATIONS

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-vision-language-models-vision-tasks/