Elamur Research⌕
☰
home / papers / 2026 / Text-Guided Multi-Scale Frequency Representation Adaptation

Text-Guided Multi-Scale Frequency Representation Adaptation

Paper · 2026 · Weicai Yan, Xinhua Ma, Wang Lin et al.

Summary

Parameter-efficient tuning of vision-language models such as CLIP usually works on spatial features. These features carry a lot of redundancy, and most methods apply one uniform adjustment that ignores scale. The authors propose FreqAdapter, which moves CLIP visual and text embeddings into the frequency domain with a DCT. There, text-conditioned modules refine the visual coefficients at several spatial scales, and an IDCT maps the result back.

The adapter has two lightweight parts. MGFA is a global bottleneck projection. MCFA predicts scale-and-shift modulation from the text spectrum. Both operate on progressively downsampled token grids, and their outputs are averaged. Training uses the CLIP contrastive loss on MS COCO 2017. The trained module can be plugged into CLIP-based systems such as LLaVA without changing the backbone.

On COCO retrieval, FreqAdapter beats CoOp, MaPLe, CLIP-Adapter and MMA. It also transfers zero-shot to Flickr30K, converges within one epoch and adds almost no compute. This suggests spectral adaptation is a cheap and effective alternative to spatial PEFT.

Key points

  • Motivation from a probing analysis: keeping only the lowest 25.8% of DCT components (198/768) already reconstructs embeddings with a cosine similarity of 0.5. Keeping 64.5% gives over 0.8, and 81.5% gives over 0.9.
  • COCO 2017 results with CLIP-B/16:
    • I2T R@1 rises from 51.82 (frozen CLIP) to 57.96, ahead of CLIP-Adapter (56.30) and MMA (55.80).
    • T2I R@1 rises from 32.65 to 43.30.
  • Zero-shot Flickr30K Val: I2T R@1 is 86.80, compared with 85.30 for the baseline.
  • Cost: about 476.4k extra parameters (0.11%) and 362.6 GFLOPs, against 362.5 GFLOPs for plain CLIP. MaPLe needs 798.7k parameters.
  • Scope: backbones tested are CLIP-B/16, CLIP-L/14 and CLIP-L/14-336, the last being the LLaVA 1.5 encoder. The adapter is also integrated into LLaVA for VQA. Code is available at github.com/Kelvin-ywc/FreqAdapter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Weicai Yan, Xinhua Ma, Wang Lin and Tao Jin. 2026. Text-Guided Multi-Scale Frequency Representation Adaptation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1108

@inproceedings{yan2026text,
  title = {{Text-Guided Multi-Scale Frequency Representation Adaptation}},
  author = {Weicai Yan, Xinhua Ma, Wang Lin and Tao Jin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1108},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1108}
}

References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Xize Cheng, Dongjie Fu, Chenyuhao Wen, Shannon Yu, Zehan Wang, Shengpeng Ji, Siddhant Arora, Tao Jin, Shinji Watanabe, and Zhou Zhao. 2025. AHa-bench: Benchmarking audio hallucinations in large audio-language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Data… find
  3. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
  4. Dongjie Fu, Xize Cheng, Xiaoda Yang, Wang Hanting, Zhou Zhao, and Tao Jin. 2024. Boosting speech recognition robustness to modality-distortion with contrast-augmented prompts. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3838–3847. find
  5. Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2024. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581–595. find
  6. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, and 1 others. 2025. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062. — Seed1.5-VL Technical Report
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  8. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. Openclip. If you use this software, please cite it as below. find
  9. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR.
  10. Can Jin, Ying Li, Mingyu Zhao, Shiyu Zhao, Zhenting Wang, Xiaoxiao He, Ligong Han, Tong Che, and Dimitris N. Metaxas. 2025. Lor-VP: Low-rank visual prompting for efficient vision model adaptation. In The Thirteenth International Conference on Learning Representations. find
  11. Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113–19122. find
  12. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  13. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  14. Ke Li, Di Wang, Zhangyuan Hu, Wenxuan Zhu, Shaofeng Li, and Quan Wang. 2024b. Unleashing channel potential: Space-frequency selection convolution for sar object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17323–17332. find
  15. Qilei Li, Wenhao Song, Mingliang Gao, Wenzhe Zhai, Qiang Zhou, and Zhao Huang. 2025. Towards text-refereed multi-modal image fusion by cross-modality interaction. Signal Processing, 237:110073. find
  16. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pag…
  17. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.
  18. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. find
  19. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning.
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  21. Badri N Patro, Vinay P Namboodiri, and Vijay Srinivas Agneeswaran. 2023. Spectformer: Frequency and attention is what you need in a vision transformer. arXiv preprint arXiv:2304.06446. — [arXiv:2304.06446](https://arxiv.org/abs/2304.06446) find
  22. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649… find
  23. Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, and Tao Mei. 2024. Boosting diffusion models with moving average sampling in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8911–8920. find
  24. Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. 2024. Federated text-driven prompt generation for vision-language models. In The Twelfth International Conference on Learning Representations. find
  25. Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML.
  26. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  27. Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. 2025. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1316–1336. find
  28. Hui Sun, Long Lv, Pingping Zhang, Tongdan Tang, Feng Tian, Weibing Sun, and Huchuan Lu. 2025. Spatial-frequency enhanced mamba for multi-modal image fusion. IEEE Transactions on Image Processing, 34:7684–7696. find
  29. Jiangtong Tan, Jie Huang, Naishan Zheng, Man Zhou, Keyu Yan, Danfeng Hong, and Feng Zhao. 2024. Revisiting spatial-frequency information integration from a hierarchical perspective for panchromatic and multi-spectral image fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt… find
  30. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.
  31. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100.
  32. Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. 2025. Visual instance-aware prompt tuning. arXiv preprint arXiv:2507.07796. — [arXiv:2507.07796](https://arxiv.org/abs/2507.07796) find
  33. Zequn Xie, Chuxin Wang, Yeqiang Wang, Sihang Cai, Shulei Wang, and Tao Jin. 2025. Chat-driven text generation and interaction for person retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5259–5270.
  34. Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. 2020. Learning in the frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  35. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023a. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.
  36. Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2023b. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36:24993–25006. find
  37. Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiaohua Xie. 2024. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23826–23837. find
  38. Runpeng Yu, Weihao Yu, and Xinchao Wang. 2024. Attention prompting on image for large vision-language models. In European Conference on Computer Vision, pages 251–268. Springer. find
  39. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
  40. Mohammad Reza Zarei, Abbas Akkasi, and Majid Komeili. 2025. Dual adapter tuning of vision– language models using large language models. International Journal of Computational Intelligence Systems, 18(1):109. find
  41. Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg, Ying Nian Wu, and Dongfang Liu. 2024. Visual fourier prompt tuning. Advances in Neural Information Processing Systems, 37:5552–5585. find
  42. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512. embedding, producing the textual embeddings E l . These embeddings,…
  43. Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930. find
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  45. The modal information is encoded as a one-dimensional tensor. The Discrete Cosine Transform (DCT) transforms this signal from the spatial domain to the frequency domain. For a signal E ∈ R D , the DCT is applied as follows: X[k] = α(k) find
  46. Hongchao Zhou and Shunbo Hu. 2024. Dual-attention frequency fusion at multi-scale for joint segmentation and deformable medical image registration. arXiv preprint arXiv:2409.19658. — [arXiv:2409.19658](https://arxiv.org/abs/2409.19658) find
  47. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  48. Discrete Cosine Transform n=0 ! π (2n + 1) k , E[n] cos 2D k = 0, 1, . . . , D − 1, where X is the frequency domain data, and α(k) are normalization factors given by:  q  1 , D α(k) = q  2 , D Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision-langua… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-text-guided-multi-scale-frequency/