Elamur Research⌕
☰
home / papers / 2026 / microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification

microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification

Paper · 2026 · Sathira Silva, Eman Ali, Chetan Arora et al.

Summary

The paper tackles adapting CLIP to fine-grained image classification without any target labels. In this setting, pseudo-labels depend on small local details that CLIP's global [CLS] token tends to overlook. Earlier unsupervised adaptation (UA) methods such as LaFTer and DPA enrich the text side with LLM descriptions but still classify from [CLS].

microCLIP is a self-training framework that adapts both modalities:

  • SOAP inside TokenFusion. Saliency-Oriented Attention Pooling uses Normalized Cut over penultimate-layer patch tokens to find salient regions. Their average then serves as the query for attention pooling, which yields a fine-grained [FG] token. TokenFusion combines this token with [CLS].
  • Two-headed classifier. Both heads are built from GPT-3 class descriptions. One stays frozen and the other is learnable.
  • Dynamic Knowledge Aggregation. This pseudo-labeler blends fixed multi-view CLIP/LLM priors with the evolving TokenFusion predictions.

Only layer norms and a small head are trained. Across 13 benchmarks, microCLIP beats the best previous UA baseline by 2.90 points on average. This shows that CLIP holds usable fine-grained signal that cheap, label-free adaptation can recover.

Key points

  • Average gain of +2.90 percentage points over the strongest prior UA baseline (e.g., DPA, LaFTer) across 13 classification datasets, including Birdsnap, RESISC, Stanford Cars, DTD, Flowers, FGVC, Pets and UCF101.
  • The multi-view prior needs only N = 8 crops, about 8 times fewer than the training-free WCA method. Accuracy plateaus beyond that point while compute keeps rising. The DKA weight γ = 0.5 is used for all datasets.
  • In the fusion ablation, averaged over six datasets, global-only logits reach 17.26%, local-only logits 57.84%, and symmetric coarse-fine fusion (λ = 0.5) 64.27%.
  • The SOAP query outperforms naive token averaging, random token selection, and plain NCut averaging as ways to build the [FG] token.
  • Limitation: gains are smaller where class evidence is spread across the whole image rather than in local parts, as with textures in DTD and some flower classes. The fixed 1:1 coarse-fine fusion weight is identified as the cause. Code is available at github.com/sathiiii/microCLIP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Sathira Silva, Eman Ali, Chetan Arora and Muhammad Haris Khan. 2026. microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1664

@inproceedings{silva2026microclip,
  title = {{microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification}},
  author = {Sathira Silva, Eman Ali, Chetan Arora and Muhammad Haris Khan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1664},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1664}
}

References

52 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eman Ali, Sathira Silva, and Muhammad Haris Khan. 2025. Dpa: Dual prototypes alignment for unsupervised adaptation of vision-language models. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6083–6093. IEEE. find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  3. Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. 2014. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2011–2018. find
  4. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. In European conference on computer vision, pages 446–461. Springer. find
  5. Gong Cheng, Junwei Han, and Xiaoqiang Lu. 2017. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883. find
  6. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613. find
  7. Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703. find
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE. find
  9. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904– 4916. PMLR.
  10. Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113–19122. find
  11. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561. find
  12. Daniel Kressner, Yuxin Ma, and Meiyue Shao. 2023. A mixed precision lobpcg algorithm. Numerical Algorithms, 94(4):1653–1671. find
  13. Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario. find
  14. Marc Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. 2024. Gallop: Learning global and local prompts for vision-language models. In European Conference on Computer Vision, pages 264–282. Springer. find
  15. Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, and Feng Liu. 2024. Visual-text cross alignment: Refining the similarity score in vision-language models. In International Conference on Machine Learning. find
  16. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888– 12900. PMLR.
  17. Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  18. Junnan Li, Silvio Savarese, and Steven CH Hoi. 2023. Masked unsupervised self-training for label-free image classification. International Conference on Learning Representations. find
  19. Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE. find
  20. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705. find
  21. Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. 2021. Vissl. https://github.com/facebookresearch/vissl. ↗
  22. Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. 2022b. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. International Conference on Learning Representations. find
  23. Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, and 1 others. 2024. Reclip: Refine contrastive language image pre-training with source free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision… find
  24. Yuqi Lin, Minghao Chen, Kaipeng Zhang, Hengjia Li, Mingming Li, Zheng Yang, Dongqin Lv, Binbin Lin, Haifeng Liu, and Deng Cai. 2024. Tagclip: A local-to-global framework to enhance open-vocabulary multi-label classification of clip without training. In Proceedings of the AAAI Conference on Artificia… find
  25. Tony Huang, Jack Chu, and Fangyun Wei. 2022. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649. — [arXiv:2204.03649](https://arxiv.org/abs/2204.03649) find
  26. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  27. Jinda Lu, Shuo Wang, Yanbin Hao, Haifeng Liu, Xiang Wang, and Meng Wang. 2024. Rethinking visual content refinement in low-shot clip adaptation. arXiv preprint arXiv:2407.14117. — [arXiv:2407.14117](https://arxiv.org/abs/2407.14117) find
  28. Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151. — [arXiv:1306.5151](https://arxiv.org/abs/1306.5151) find
  29. Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. 2023. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. Advances in Neural Information Processing Systems, 36:5765–5777. find
  30. Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE. find
  31. OpenAI. 2024. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/. Accessed: 2026-04-14. ↗
  32. Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. 2012. Cats and dogs. IEEE Conference on Computer Vision and Pattern Recognition, pages 3498– 3505. find
  33. Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. 2023. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691–15701. find
  34. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  35. Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888–905. find
  36. Gyungin Shin, Samuel Albanie, and Weidi Xie. 2022. Unsupervised salient object detection with spectral cluster voting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3971– 3980. find
  37. Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobeckỳ, Éloi Zablocki, and Patrick Pérez. 2023. Unsupervised object localization: Observing the background to discover objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3176– 3186. find
  38. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638–15650. find
  39. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. Center for Research in Computer Vision. find
  40. Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou. 2023. Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models. In International Conference on Machine Learning, pages 33816–33832. PMLR. find
  41. Feng Wang, Jieru Mei, and Alan Yuille. 2024. Sclip: Rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, pages 315–332. Springer. find
  42. Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. 2023. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12):15790–15… find
  43. Xinye Wanyan, Sachith Seneviratne, Shuchang Shen, and Michael Kirley. 2023. Dino-mc: Self-supervised contrastive learning for remote sensing imagery with multi-sized local crops. arXiv preprint arXiv:2303.06670, 2(6):26. — [arXiv:2303.06670](https://arxiv.org/abs/2303.06670) find
  44. Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE. find
  45. Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, and Stephan Alaniz. 2025. Flair: Vlm with fine-grained language-informed image representations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24884–24894. find
  46. Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. 2024a. Demystifying clip data. International Conference on Learning Representations. find
  47. Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. 2024b. Demystifying clip data. International Conference on Learning Representations. find
  48. Maxime Zanella and Ismail Ben Ayed. 2024. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593–1603. find
  49. Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pages 493–510. Springer. find
  50. Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. 2024. Dreamlip: Language-image pre-training with long captions. In European Conference on Computer Vision, pages 73–90. Springer. find
  51. Chong Zhou, Chen Change Loy, and Bo Dai. 2022. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696–712. Springer. find
  52. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2021. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337 – 2348. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/silva-2026-microclip-unsupervised-clip-adaptation-coarse/